Skip to main content

Nutch crawler and integration with Solr

Before moving ahead with this article, I assume you have Solr installed and running. If you would like to install Solr on windows, mac or via docker, please read Setup a Solr instance.

There are several ways to install nutch which you can read from Nutch tutorial, however I have written this article for those who would like to install nutch using docker. I tried finding help on google but could not find any help for nutch installation using docker and spent good amount of time fixing issues specific to it. Therefore I have written this article to help and save time of other developers.

Install nutch using docker-

1. Pull docker image of nutch using below command,
    > docker pull apache/nutch
2. Once image is pulled, run the container,
    > docker run -t -i -d --name nutchcontainer apache/nutch /bin/bash
3. You should be able to enter in the container and see bash prompt,
    > bash-5.1# 

Let's setup few important settings now-

1. Goto bin folder, 
    > bash-5.1# cd /nutch/bin
        you should find nutch and crawl scripts in bin folder.
2. Check nutch is installed by running below command,
    > bash-5.1# ./nutch
        you should get nutch details as output.

3. Create a new folder where we will add our URLs for crawl,
    > bash-5.1# mkdir urls
    > bash-5.1# touch seed.txt
    > bash-5.1# vi seed.txt

         add urls in seed.txt which you would like to crawl.

4. seed.txt file (in this case, I have added two urls to crawl) should look like this,

5. Modify nutch-site.xml file,
    > bash-5.1# cd ../conf
    > bash-5.1# vi nutch-site.xml
    Remove existing lines and add following lines in it,
    <?xml version="1.0"?>
    <configuration>
         <property>
          <name>http.agent.name</name>
          <value>nutch-solr-integration</value>
         </property>
         <property>
          <name>generate.max.count</name>
          <value>10</value>
         </property>
         <property>
          <name>generate.max.per.host</name>
          <value>10</value>
         </property>
         <property>
          <name>plugin.includes</name>
          <value>protocol-http|urlfilter-regex|parse-html|index-(basic|anchor)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer
         </property>
    </configuration>

6. Run below command to inject the urls for crawling,
    > bash-5.1# cd ../bin 
    > bash-5.1# ./nutch inject crawlerdb urls
        It should inject the urls and show you a success message. Along with it, a crawldb will also be                    created. 

7. Now run, below command to crawl, create segments and invertedlinks,

    > bash-5.1# ./crawl --num-threads 3 -s urls crawldb 2

    this will start crawling of urls with 3 consecutive threads and iterate it 2 times.


8. After successfull run, you will notice, a crawldb, a segment and a linkdb folders are created.  

Integrate it with Solr - 

1. Modify index-writers.xml file,
    > bash-5.1# cd ../conf
    > bash-5.1# vi index-writers.xml
     Update solr url as shown below, where nutch_collection is collection created in Solr. 
2. Run the command to push it to Solr instance,

    > bash-5.1# ./crawl -i -D solr.server.url=http://solr.bajajsumit.com:8983/solr/nutch_collection crawldb 1


3. After successful completion, records should show in Solr instance->nutch_collection


Please feel free to ask for any help. Keep learning and building the community strong. 


Comments

  1. Best Casinos Near Washington DC 2021 - Mapyro
    Find the 청주 출장샵 best casinos near Washington DC 2021 with mapyro 남양주 출장마사지 users' 경주 출장안마 ratings, mapyro users comments and more. Riverview Casino 군산 출장마사지 Resort 익산 출장마사지 Casino, D

    ReplyDelete
  2. Thank you for this wonderful guide! I have a question though: what to do when received the error "No User-Agent string set". I can't get past this error

    2022-04-29 19:19:42,438 ERROR o.a.n.p.h.a.HttpBase [main] No User-Agent string set (http.agent.name)!
    2022-04-29 19:19:42,560 WARN o.a.n.p.PluginRepository [main] Could not find org.apache.nutch.protocol.http.Http
    java.lang.RuntimeException: Agent name not configured!

    ReplyDelete

Post a Comment

Popular posts from this blog

Cannot alter the login 'sa', because it does not exist or you do not have permission.

Working on projects, it can happen that 'sa' account gets locked. If it is on local machine OR development boxes, onus would be on you to fix it. If scripts and SQL steps are not working, this might help you fixing the issue. Steps to unlock 'sa' account and resetting the password. 1. Open SQL Server Configuration Manager 2. Select SQL Server Services -> 'SQL Server' service. 3. Right click on 'SQL Server' service and click on "Startup Parameters". For 2008, server "Startup Parameters" are inside Advanced tab.   4. Add '-m' in startup parameters as shown above and click on 'Add'. This will put SQL server into 'Single User Mode' and local admin will have 'Super User' rights. For 2008, server you have to add ':-m' in the last of the existing query. 5. Save the settings and Restart the service. 6. Now open the SQL Server Management Studio and connect to database using 'Windows A

Could not load file or assembly 'Microsoft.Web.Infrastructure'

Could not load file or assembly 'Microsoft.Web.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. What 'Micorosoft.Web.Infrastructure' does? This dll lets HTTP modules register at run time. Solution to above problem: Copy 'Micorosoft.Web.Infrastructure' dll in bin folder of your project and this problem should be resolved. If you have .Net framework installed on machine, this dll should be present on it. You can search for this dll and copy it in your active project folder.   Alternatively,  you can install this dll using nuget package manager PM> Install-Package Microsoft.Web.Infrastructure -Version 1.0.0 Happy coding!!

Git merge from one repo to another repo

This blog is for those who are looking for merging code from one repo to another repo. Why I will merge code from one repo to another? I forked from one git repo( may be some public git repo ) and did some cutomization on existing code. Occasionaly, features are being introduced in main git branch and I would like to get all those features in my own git repo. In this scenario, I would like to merge latest changes in my git repo. Here are few simple steps which will merge code from one git repo to another- 1. Clone the repo1(source git repo) > git clone https://github.com/org/repo1.git   > git pull 2. Clone the repo2(destination git repo) > git clone https://github.com/org/repo2.git > git pull 3. Goto repo2 (destination git folder) and checkout your prefered branch > cd repo2 > git checkout master>  4. Use below command to see the remote branch associated with your destination git repo > git remote -v  origin https://github.com/org/repo2.g