Get up and running quickly with Nutch on Docker.

Apache Nutch is a highly extensible and scalable open source web crawler software project.

Nutch can run on a single machine, but gains a lot of its strength from running in a Hadoop cluster

Current configuration of this image consists of components:

• Nutch 1.x (branch "master")
• Default runtime user nutch (not root)
• NUTCH_HOME=/opt/nutch/runtime/local (sources under /opt/nutch)

• alpine:3.19

You may need to alias docker to "docker --tls" if you see errors such as:

2015/04/07 09:19:56 Post http://192.168.59.103:2376/v1.14/containers/create?name=NutchContainer: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"

The easiest way to do this:

1. alias docker="docker --tls"

1. Install Docker.

2. Build from files in this directory:

$(boot2docker shellinit | grep export) #may not be necessary
docker build -t apache/nutch .

Pull requests that change docker/Dockerfile (or the docker image smoke workflow) build this image and inject https://nutch.apache.org/ into a CrawlDb. They do not run fetch.

Nutch loads executable code from the directories configured as plugin.folders (see nutch-default.xml). For production and shared images, treat those paths as trusted: mount them read-only where possible, rebuild images to change plugins, and keep the crawl process on the image default user nutch so the filesystem cannot be abused to drop unexpected JARs or plugin.xml files into that tree. Use docker run --user root ... only for privileged debugging.

User-defined JEXL in configuration (for example index.jexl.filter, generator expressions, and hostdb.filter.expression) is evaluated in a sandboxed engine by default. The property nutch.jexl.disable.sandbox disables that protection and must not be set in untrusted environments.

If not already running, start docker

boot2docker up
$(boot2docker shellinit | grep export)

Run a container interactively (nutch and crawl are on PATH; default command is bash; process user is nutch):

docker run -t -i --name nutchcontainer apache/nutch

The previous image layout used /root/nutch_source. Bind mounts that pointed at that path must be updated to /opt/nutch.

In another terminal, attach to a running container if needed:

docker exec -it nutchcontainer /bin/bash

Nutch is located in $NUTCH_HOME and is almost ready to run. You will need to set seed URLs and update the http.agent.name configuration property in $NUTCH_HOME/conf/nutch-site.xml with your crawler's Agent Name. For additional "getting started" information checkout the Nutch Tutorial.