Nutch crawler and integration with Solr

Before moving ahead with this article, I assume you have Solr installed and running. If you would like to install Solr on windows, mac or via docker, please read Setup a Solr instance.

There are several ways to install nutch which you can read from Nutch tutorial, however I have written this article for those who would like to install nutch using docker. I tried finding help on google but could not find any help for nutch installation using docker and spent good amount of time fixing issues specific to it. Therefore I have written this article to help and save time of other developers.

Install nutch using docker-

1. Pull docker image of nutch using below command,

    > docker pull apache/nutch
2. Once image is pulled, run the container,
    > docker run -t -i -d --name nutchcontainer apache/nutch /bin/bash
3. You should be able to enter in the container and see bash prompt,
    > bash-5.1# 

Let's setup few important settings now-

1. Goto bin folder, 
    > bash-5.1# cd /nutch/bin
        you should find nutch and crawl scripts in bin folder.
2. Check nutch is installed by running below command,
    > bash-5.1# ./nutch
        you should get nutch details as output.

3. Create a new folder where we will add our URLs for crawl,
    > bash-5.1# mkdir urls
    > bash-5.1# touch seed.txt
    > bash-5.1# vi seed.txt

         add urls in seed.txt which you would like to crawl.

4. seed.txt file (in this case, I have added two urls to crawl) should look like this,

5. Modify nutch-site.xml file,

    > bash-5.1# cd ../conf

    > bash-5.1# vi nutch-site.xml

    Remove existing lines and add following lines in it,

    <?xml version="1.0"?>

    <configuration>

         <property>

          <name>http.agent.name</name>

          <value>nutch-solr-integration</value>

         </property>

         <property>

          <name>generate.max.count</name>

          <value>10</value>

         </property>

         <property>

          <name>generate.max.per.host</name>

          <value>10</value>

         </property>

         <property>

          <name>plugin.includes</name>

          <value>protocol-http|urlfilter-regex|parse-html|index-(basic|anchor)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer

         </property>

    </configuration>

6. Run below command to inject the urls for crawling,

    > bash-5.1# cd ../bin 

    > bash-5.1# ./nutch inject crawlerdb urls

        It should inject the urls and show you a success message. Along with it, a crawldb will also be                    created. 

7. Now run, below command to crawl, create segments and invertedlinks,

    > bash-5.1# ./crawl --num-threads 3 -s urls crawldb 2

    this will start crawling of urls with 3 consecutive threads and iterate it 2 times.

8. After successfull run, you will notice, a crawldb, a segment and a linkdb folders are created.  

Integrate it with Solr - 

1. Modify index-writers.xml file,

    > bash-5.1# cd ../conf

    > bash-5.1# vi index-writers.xml

     Update solr url as shown below, where nutch_collection is collection created in Solr. 

        

2. Run the command to push it to Solr instance,

    > bash-5.1# ./crawl -i -D solr.server.url=http://solr.bajajsumit.com:8983/solr/nutch_collection crawldb 1

3. After successful completion, records should show in Solr instance->nutch_collection

Please feel free to ask for any help. Keep learning and building the community strong.