Apache Nutch

Apache Nutch is a highly extensible and scalable open-source web crawler software project built on Apache Hadoop data structures for batch processing. It provides a pluggable architecture supporting custom parse filters, scoring filters, index writers, and protocol implementations. Nutch integrates with Apache Solr and Elasticsearch for full-text search and exposes a REST API for managing crawl jobs, configurations, seed lists, and database queries. Governed by the Apache Software Foundation under the Apache License 2.0.

Apache Nutch publishes 7 APIs on the APIs.io network, including Admin API, Configuration API, Database API, and 4 more. Tagged areas include Web Crawler, Indexing, Search, Apache, and Java.

The Apache Nutch catalog on APIs.io includes 1 JSON-LD context and 2 Spectral governance rulesets.

Apache Nutch’s developer surface includes authentication, documentation, getting-started guide, FAQ, release notes, support, Stack Overflow tag, and 15 more developer resources.

📡 Source on GitHub

42.7/100 developing ▬ flat Agent 22/100 agent aware open core · Apache-2.0 Full breakdown ↓
scored 2026-09-08 · rubric v0.20.0

AccessFreemiumSelf serve

1 APIs 10 Features 6 Use Cases

Web CrawlerIndexing Search Apache Java Hadoop Open-Source

Kin Score

Kin Score How this is scored →
scored 2026-09-08 · rubric v0.20.0

Composite quality — 42.7/100 · developing

Contract Quality 13.4 / 20

Developer Ergonomics 6.7 / 16

Access Clarity 4.2 / 16

Operational Transparency 3.8 / 10

Contract Governance 1.3 / 10

Discoverability 4.7 / 8

Open Source Surface 5.0 / 10

Create-or-Update Ergonomics 0.0 / 10

Agent readiness — 22/100 · agent aware

Machine-Readable Contract 18 / 18

Agentic Access Contract 10 / 10

Documented Reversibility 0 / 6

MCP Server 0 / 12

Machine-Readable Auth 10 / 10

Idempotency 0 / 9

Stable Error Semantics 0 / 8

Request/Response Examples 7 / 7

Rate-Limit Signaling 7 / 7

Typed Event Surface 0 / 6

Agent Skills 0 / 5

Well-Known Catalog 0 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Delegated User Identity 0 / 6

Protected Resource Metadata 0 / 5

Registration Without a Human 0 / 6

Agentic Commerce Well-Known Document 0 / 5

Open Source Surface applies to this provider. This product is open source and we read its repository directly, so Open Source Surface carries 10 points of the composite. It is scored from what the repository actually publishes — a security policy, a contribution guide, a release history, a code of conduct — read live from the provider rather than inferred from our own catalog pointers. This facet adds; nothing was taken away to make room for it. An open-source project is not excused from the commercial facets, because exemption would strip it of the points it does earn. If we have the wrong repository, or this product is not open source, say so on your provider repo and we will drop the facet rather than have you publish against it.

Create-or-Update Ergonomics applies to this provider. This API accepts writes, so it carries 10 points of the composite. It is scored from the published contracts themselves: whether a caller can create-or-update in one call, whether the write accepts a key the caller already holds, and whether the response says which branch ran. Without that, every write needs a search-and-branch in front of it, and the first time that check is skipped a duplicate record is created. Scored against the observed mean rather than raw — a provider at the catalog average is unchanged by this facet, not penalised by it.

The six quality facets above are damped to 80 points between them, because the conditional facet above carries the other 20. That is why each facet's contribution is shown against a damped maximum: raising a quality facet moves the composite by 80% of its nominal weight, not 100%. The full arithmetic is at apis.io/rating/.

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from github.com/api-evangelist/apache-nutch: open an issue to ask a question, or submit a pull request to add artifacts. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

APIs 7

Individual APIs this provider publishes, each with its own machine-readable definition.

Apache Nutch Admin API

Server administration operations

Apache Nutch Configuration API

Manage Nutch configurations

Apache Nutch Database API

Query the CrawlDB and FetchDB

Apache Nutch Job API

Manage crawl jobs

Apache Nutch Reader API

Read sequence files and webgraph data

Apache Nutch Seed API

Manage seed URL lists

Apache Nutch Services API

Auxiliary service operations such as CommonCrawl data dumps

POSTMAN

Pricing Plans 1

Published pricing tiers and plan structures.

Apache Nutch Plans Pricing

3 plans

PLANS

Rate Limits 1

Documented rate limits and quota policies.

Apache Nutch Rate Limits

5 limits

RATE LIMITS

FinOps 1

Cost, billing, and metering signals for API financial operations.

Apache Nutch Finops

FINOPS

Features 10

Notable capabilities this provider offers.

Scalable Batch Crawling

Leverages Apache Hadoop data structures for distributed, large-scale web crawling batch processing.

Pluggable Architecture

Extensible plugin system supporting custom parse filters, scoring filters, index writers, protocol plugins, and URL filters.

REST API for Crawl Management

Full REST API for managing crawl jobs, configurations, seed lists, CrawlDB/FetchDB queries, and sequence file readers.

Full-Text Search Integration

Built-in index writers for Apache Solr and Elasticsearch to enable full-text search over crawled content.

Apache Tika Parsing

Uses Apache Tika for parsing a wide variety of document formats during the crawl pipeline.

Duplicate Detection

Built-in deduplication support to identify and remove duplicate content from the crawl database and search index.

Configurable URL Filtering

Regex-based and custom URL filter plugins to control crawl scope and exclusions.

Incremental Crawling

Supports multi-round incremental crawling workflows to keep the crawl database fresh.

CommonCrawl Export

Service operations for exporting crawl data in CommonCrawl-compatible formats.

HTTP Authentication Support

Configurable HTTP authentication schemes for crawling password-protected sites.

Scroll for all 10

Semantic Vocabularies 1

JSON-LD contexts and semantic vocabularies used across these APIs.

Apache Nutch Context

16 classes · 35 properties

JSON-LD

Spectral Rules 2

Spectral governance rulesets for linting and validating these APIs.

Apache Nutch API Rules

5 rules · 3 warnings 2 info

SPECTRAL

Apache Nutch API Rules

39 rules · 12 errors 24 warnings 3 info

SPECTRAL

0 properties

JSON STRUCTURE

Scroll for all 16

Recommended x-agentic-access execution contracts for AI agents.

Apache Nutch Agentic Access

24 operations · 10 acting

AGENTIC

Use Cases 6

What developers build with this provider.

Enterprise Search

Build enterprise search engines over internal or external web content using Nutch as the crawler and Solr/Elasticsearch as the search backend.

Research Data Collection

Academic and research teams use Nutch for large-scale systematic web data collection and indexing.

Intranet Document Search

Crawl and index intranet sites, wikis, and document repositories for internal enterprise search.

Web Archive Creation

Create structured web archives compatible with CommonCrawl format for long-term data preservation.

SEO and Content Monitoring

Monitor web content changes, track competitor sites, and analyze web structure at scale.

Custom Data Extraction Pipelines

Build custom extraction pipelines using Nutch plugin architecture for targeted data acquisition tasks.

Integrations 5

Pre-built integrations with other platforms and tools.

Apache Solr

Native index writer plugin for indexing crawled content into Apache Solr for full-text search.

Elasticsearch

Index writer plugin for sending crawled content to Elasticsearch clusters.

Apache Hadoop

Core dependency providing distributed storage and processing via HDFS and MapReduce.

Apache Tika

Used for content detection and extraction from a wide range of document formats during parsing.

SolrCloud

Support for SolrCloud distributed search clusters for scalable indexing.

Resources

Get Started 1

Portal, sign-up, and the first successful call

Nutch Tutorial

GettingStarted

Documentation 1

Reference material describing how the API behaves

Apache Nutch Documentation

Documentation

Agent Surfaces 1

MCP servers, agent skills, and machine-readable catalogs

AgenticAccess

AgenticAccess

Design & Contract 3

Pagination, idempotency, versioning, errors, and events

Apache Nutch Spectral Rules

SpectralRules

Apache Nutch Vocabulary

Vocabulary

Apache Nutch JSON-LD Context

JSONLD

Build 4

SDKs, sample code, and the tooling you integrate with

CodeOfConduct

CodeOfConduct

PostmanWorkspace

PostmanWorkspace

Apache Nutch GitHub Repository

GitHubRepository

Apache Software Foundation GitHub

GitHubOrganization

Access & Security 4

Authentication, authorization, and security posture

SecurityPolicy

SecurityPolicy

VulnerabilityDisclosure

VulnerabilityDisclosure

DomainSecurity

DomainSecurity

Authentication

Authentication

Learn 1

Tutorials, courses, talks, and written guidance

Apache Nutch Tutorials

Tutorials

Operate 4

Status, limits, changes, and where to get help

Apache Nutch FAQs

FAQ

Nutch Release Notes

ReleaseNotes

Mailing Lists

Support

Nutch on Stack Overflow

StackOverflow

Commercial 2

Pricing, plans, and the legal terms of use

License

License

Apache License 2.0

TermsOfService

Company 1

The organization behind the API

Blog

Blog

Source (apis.yml)

aid: apache-nutch
name: Apache Nutch
description: Apache Nutch is a highly extensible and scalable open-source web crawler software project built on Apache Hadoop
  data structures for batch processing. It provides a pluggable architecture supporting custom parse filters, scoring filters,
  index writers, and protocol implementations. Nutch integrates with Apache Solr and Elasticsearch for full-text search and
  exposes a REST API for managing crawl jobs, configurations, seed lists, and database queries. Governed by the Apache Software
  Foundation under the Apache License 2.0.
type: Index
deliveryModel:
  model: open-core
  license: Apache-2.0
  open_source: true
  commercial: true
  callable_host: false
  label: Open core · an OSS project plus a commercial hosted product
  confidence: high
  source:
  - license
  - openapi
  - pricing
  generated: '2026-08-28'
  method: derived
accessModel:
  pricing: freemium
  onboarding: self-serve
  trial: false
  try_now: false
  public: false
  label: Freemium · Self-serve signup
  confidence: medium
  try_now_blocked_by: loopback
  source:
  - plans
  - authentication
  - security
  generated: '2026-09-03'
  method: derived
position: Consuming
access: 3rd-Party
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/apache-nutch.png
tags:
- Web Crawler
- Indexing
- Search
- Apache
- Java
- Hadoop
- Open-Source
tags_raw:
- Web Crawler
- Indexing
- Search
- Apache
- Java
- Hadoop
- Open Source
created: '2026-03-16'
modified: '2026-05-19'
url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/apis.yml
specificationVersion: '0.23'
apis:
- aid: apache-nutch:apache-nutch-admin-api
  name: Apache Nutch Admin API
  description: Server administration operations
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Admin
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-admin-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-configuration-api
  name: Apache Nutch Configuration API
  description: Manage Nutch configurations
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Configuration
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-configuration-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-database-api
  name: Apache Nutch Database API
  description: Query the CrawlDB and FetchDB
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Database
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-database-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-job-api
  name: Apache Nutch Job API
  description: Manage crawl jobs
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Job
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-job-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-reader-api
  name: Apache Nutch Reader API
  description: Read sequence files and webgraph data
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Reader
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-reader-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-seed-api
  name: Apache Nutch Seed API
  description: Manage seed URL lists
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Seed
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-seed-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-services-api
  name: Apache Nutch Services API
  description: Auxiliary service operations such as CommonCrawl data dumps
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Services
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-services-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
common:
- type: SecurityPolicy
  url: https://github.com/apache/nutch/blob/master/SECURITY.md
- type: CodeOfConduct
  url: https://github.com/apache/.github/blob/main/.github/CODE_OF_CONDUCT.md
- type: License
  name: Apache-2.0
  url: https://github.com/apache/nutch/blob/master/LICENSE
- type: PostmanWorkspace
  url: https://www.postman.com/kinlaneapi/apache-nutch/overview
- type: AgenticAccess
  url: agentic-access/apache-nutch-agentic-access.yml
- type: VulnerabilityDisclosure
  url: security/apache-nutch-vulnerability-disclosure.yml
- type: DomainSecurity
  url: security/apache-nutch-domain-security.yml
- type: Authentication
  url: authentication/apache-nutch-authentication.yml
- type: GitHubRepository
  url: https://github.com/apache/nutch
  title: Apache Nutch GitHub Repository
- type: GitHubOrganization
  url: https://github.com/apache
  title: Apache Software Foundation GitHub
- type: Documentation
  url: https://nutch.apache.org/documentation/
  title: Apache Nutch Documentation
- type: GettingStarted
  url: https://cwiki.apache.org/confluence/display/NUTCH/NutchTutorial
  title: Nutch Tutorial
- type: Tutorials
  url: https://nutch.apache.org/documentation/tutorials/
  title: Apache Nutch Tutorials
- type: FAQ
  url: https://nutch.apache.org/documentation/faqs/
  title: Apache Nutch FAQs
- type: ReleaseNotes
  url: https://github.com/apache/nutch/blob/master/CHANGES.md
  title: Nutch Release Notes
- type: TermsOfService
  url: https://www.apache.org/licenses/LICENSE-2.0
  title: Apache License 2.0
- type: Support
  url: https://nutch.apache.org/community/mailing-lists/
  title: Mailing Lists
- type: StackOverflow
  url: https://stackoverflow.com/questions/tagged/nutch
  title: Nutch on Stack Overflow
- type: SpectralRules
  url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/rules/apache-nutch-spectral-rules.yml
  title: Apache Nutch Spectral Rules
- type: Vocabulary
  url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/vocabulary/apache-nutch-vocabulary.yaml
  title: Apache Nutch Vocabulary
- type: JSONLD
  url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-ld/apache-nutch-context.jsonld
  title: Apache Nutch JSON-LD Context
- type: Features
  data:
  - name: Scalable Batch Crawling
    description: Leverages Apache Hadoop data structures for distributed, large-scale web crawling batch processing.
  - name: Pluggable Architecture
    description: Extensible plugin system supporting custom parse filters, scoring filters, index writers, protocol plugins,
      and URL filters.
  - name: REST API for Crawl Management
    description: Full REST API for managing crawl jobs, configurations, seed lists, CrawlDB/FetchDB queries, and sequence
      file readers.
  - name: Full-Text Search Integration
    description: Built-in index writers for Apache Solr and Elasticsearch to enable full-text search over crawled content.
  - name: Apache Tika Parsing
    description: Uses Apache Tika for parsing a wide variety of document formats during the crawl pipeline.
  - name: Duplicate Detection
    description: Built-in deduplication support to identify and remove duplicate content from the crawl database and search
      index.
  - name: Configurable URL Filtering
    description: Regex-based and custom URL filter plugins to control crawl scope and exclusions.
  - name: Incremental Crawling
    description: Supports multi-round incremental crawling workflows to keep the crawl database fresh.
  - name: CommonCrawl Export
    description: Service operations for exporting crawl data in CommonCrawl-compatible formats.
  - name: HTTP Authentication Support
    description: Configurable HTTP authentication schemes for crawling password-protected sites.
- type: UseCases
  data:
  - name: Enterprise Search
    description: Build enterprise search engines over internal or external web content using Nutch as the crawler and Solr/Elasticsearch
      as the search backend.
  - name: Research Data Collection
    description: Academic and research teams use Nutch for large-scale systematic web data collection and indexing.
  - name: Intranet Document Search
    description: Crawl and index intranet sites, wikis, and document repositories for internal enterprise search.
  - name: Web Archive Creation
    description: Create structured web archives compatible with CommonCrawl format for long-term data preservation.
  - name: SEO and Content Monitoring
    description: Monitor web content changes, track competitor sites, and analyze web structure at scale.
  - name: Custom Data Extraction Pipelines
    description: Build custom extraction pipelines using Nutch plugin architecture for targeted data acquisition tasks.
- type: Integrations
  data:
  - name: Apache Solr
    description: Native index writer plugin for indexing crawled content into Apache Solr for full-text search.
  - name: Elasticsearch
    description: Index writer plugin for sending crawled content to Elasticsearch clusters.
  - name: Apache Hadoop
    description: Core dependency providing distributed storage and processing via HDFS and MapReduce.
  - name: Apache Tika
    description: Used for content detection and extraction from a wide range of document formats during parsing.
  - name: SolrCloud
    description: Support for SolrCloud distributed search clusters for scalable indexing.
- url: https://nutch.apache.org/index.xml
  type: Blog
maintainers:
- FN: Kin Lane
  email: info@apievangelist.com

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this

find_providersBrowse and filter every provider in the catalog.
get_provider_artifactsEvery artifact this provider publishes, grouped by type.
get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
get_provider_ratingPRO — composite, band, trend and facet scores.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This provider

curl "https://apis.io/api/v1/providers/apache-nutch"

All providers

curl "https://apis.io/api/v1/providers?limit=25"

Every operation they expose

curl "https://apis.io/api/v1/providers/apache-nutch/operations?limit=25"

How their score was established

curl "https://apis.io/api/v1/providers/apache-nutch/evidence"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.