Apache Nutch

Apache Nutch is a highly extensible and scalable open-source web crawler software project built on Apache Hadoop data structures for batch processing. It provides a pluggable architecture supporting custom parse filters, scoring filters, index writers, and protocol implementations. Nutch integrates with Apache Solr and Elasticsearch for full-text search and exposes a REST API for managing crawl jobs, configurations, seed lists, and database queries. Governed by the Apache Software Foundation under the Apache License 2.0.

Apache Nutch publishes 7 APIs on the APIs.io network, including Admin API, Configuration API, Database API, and 4 more. Tagged areas include Web Crawler, Indexing, Search, Apache, and Java.

The Apache Nutch catalog on APIs.io includes 1 JSON-LD context and 2 Spectral governance rulesets.

Apache Nutch’s developer surface includes authentication, documentation, getting-started guide, FAQ, release notes, support, Stack Overflow tag, and 12 more developer resources.

🌐 Visit website 📡 Source on GitHub

58.4/100 strong ▬ flat Agent 34/100 agent aware Full breakdown ↓
scored 2026-07-28 · rubric v0.6

AccessFreemiumSelf serve⚡ Free to try

7 APIs 10 Features 6 Use Cases

Web CrawlerIndexingSearchApacheJavaHadoopOpen Source

Kin Score

Kin Score How this is scored →
scored 2026-07-28 · rubric v0.6

Composite quality — 58.4/100 · strong

Contract Quality 18.6 / 25

Developer Ergonomics 8.3 / 20

Commercial Clarity 10.0 / 20

Operational Transparency 6.8 / 13

Governance 8.3 / 12

Discoverability 6.5 / 10

Agent readiness — 34/100 · agent aware

Machine-Readable Contract 18 / 18

Agentic Access Contract 10 / 10

MCP Server 0 / 12

Machine-Readable Auth 10 / 10

Idempotency 0 / 9

Stable Error Semantics 0 / 8

Request/Response Examples 7 / 7

Rate-Limit Signaling 7 / 7

Typed Event Surface 0 / 6

Agent Skills 0 / 5

Well-Known Catalog 0 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. This rating is computed from github.com/api-evangelist/apache-nutch: open an issue to ask a question, or submit a pull request to add artifacts. Want it done for you? Prioritized profiling — $2,500 →

APIs 7

Individual APIs this provider publishes, each with its own machine-readable definition.

Apache Nutch Admin API

Server administration operations

Apache Nutch Configuration API

Manage Nutch configurations

Apache Nutch Database API

Query the CrawlDB and FetchDB

Apache Nutch Job API

Manage crawl jobs

Apache Nutch Reader API

Read sequence files and webgraph data

Apache Nutch Seed API

Manage seed URL lists

Apache Nutch Services API

Auxiliary service operations such as CommonCrawl data dumps

Scroll for all 7

Postman Collections 7

Ready-to-run Postman collections for exercising this provider's APIs.

Apache Nutch REST Admin API

POSTMAN

Apache Nutch REST Admin Configuration API

POSTMAN

Apache Nutch REST Admin Database API

POSTMAN

Apache Nutch REST Admin Job API

POSTMAN

Apache Nutch REST Admin Reader API

POSTMAN

Apache Nutch REST Admin Seed API

POSTMAN

Apache Nutch REST Admin Services API

POSTMAN

Scroll for all 7

Pricing Plans 1

Published pricing tiers and plan structures.

Apache Nutch Plans Pricing

3 plans

PLANS

Rate Limits 1

Documented rate limits and quota policies.

Apache Nutch Rate Limits

5 limits

RATE LIMITS

FinOps 1

Cost, billing, and metering signals for API financial operations.

Apache Nutch Finops

FINOPS

Features 10

Notable capabilities this provider offers.

Scalable Batch Crawling

Leverages Apache Hadoop data structures for distributed, large-scale web crawling batch processing.

Pluggable Architecture

Extensible plugin system supporting custom parse filters, scoring filters, index writers, protocol plugins, and URL filters.

REST API for Crawl Management

Full REST API for managing crawl jobs, configurations, seed lists, CrawlDB/FetchDB queries, and sequence file readers.

Full-Text Search Integration

Built-in index writers for Apache Solr and Elasticsearch to enable full-text search over crawled content.

Apache Tika Parsing

Uses Apache Tika for parsing a wide variety of document formats during the crawl pipeline.

Duplicate Detection

Built-in deduplication support to identify and remove duplicate content from the crawl database and search index.

Configurable URL Filtering

Regex-based and custom URL filter plugins to control crawl scope and exclusions.

Incremental Crawling

Supports multi-round incremental crawling workflows to keep the crawl database fresh.

CommonCrawl Export

Service operations for exporting crawl data in CommonCrawl-compatible formats.

HTTP Authentication Support

Configurable HTTP authentication schemes for crawling password-protected sites.

Scroll for all 10

Semantic Vocabularies 1

JSON-LD contexts and semantic vocabularies used across these APIs.

Apache Nutch Context

16 classes · 35 properties

JSON-LD

Spectral Rules 2

Spectral governance rulesets for linting and validating these APIs.

Apache Nutch API Rules

5 rules · 3 warnings 2 info

SPECTRAL

Apache Nutch API Rules

39 rules · 12 errors 24 warnings 3 info

SPECTRAL

JSON Schema 16

Standalone JSON Schema definitions for this provider's data models.

ChildNode

2 properties

JSON SCHEMA

DbQuery

4 properties

JSON SCHEMA

FetchNodeDbInfo

4 properties

JSON SCHEMA

JobConfig

5 properties

JSON SCHEMA

JobInfo

8 properties

JSON SCHEMA

JobType

0 properties

JSON SCHEMA

LinkSchema

6 properties

JSON SCHEMA

NodeSchema

6 properties

JSON SCHEMA

NutchConfig

3 properties

JSON SCHEMA

NutchServerInfo

4 properties

JSON SCHEMA

ReaderConfig

1 properties

JSON SCHEMA

SeedList

4 properties

JSON SCHEMA

SeedUrl

2 properties

JSON SCHEMA

ServiceConfig

3 properties

JSON SCHEMA

ServiceInfo

1 properties

JSON SCHEMA

State

0 properties

JSON SCHEMA

Scroll for all 16

JSON Structure 16

JSON Structure definitions describing this provider's data shapes.

Apache Nutch Child Node Structure

2 properties

JSON STRUCTURE

Apache Nutch Db Query Structure

4 properties

JSON STRUCTURE

Apache Nutch Fetch Node Db Info Structure

4 properties

JSON STRUCTURE

Apache Nutch Job Config Structure

5 properties

JSON STRUCTURE

Apache Nutch Job Info Structure

8 properties

JSON STRUCTURE

Apache Nutch Job Type Structure

0 properties

JSON STRUCTURE

Apache Nutch Link Schema Structure

6 properties

JSON STRUCTURE

Apache Nutch Node Schema Structure

6 properties

JSON STRUCTURE

Apache Nutch Nutch Config Structure

3 properties

JSON STRUCTURE

Apache Nutch Nutch Server Info Structure

4 properties

JSON STRUCTURE

Apache Nutch Reader Config Structure

1 properties

JSON STRUCTURE

Apache Nutch Seed List Structure

4 properties

JSON STRUCTURE

Apache Nutch Seed Url Structure

2 properties

JSON STRUCTURE

Apache Nutch Service Config Structure

3 properties

JSON STRUCTURE

Apache Nutch Service Info Structure

1 properties

JSON STRUCTURE

Apache Nutch State Structure

0 properties

JSON STRUCTURE

Scroll for all 16

Examples 16

Example request and response payloads for these APIs.

Apache Nutch Child Node Example

2 fields

EXAMPLE

Apache Nutch Db Query Example

4 fields

EXAMPLE

Apache Nutch Fetch Node Db Info Example

4 fields

EXAMPLE

Apache Nutch Job Config Example

5 fields

EXAMPLE

Apache Nutch Job Info Example

8 fields

EXAMPLE

Apache Nutch Job Type Example

0 fields

EXAMPLE

Apache Nutch Link Schema Example

6 fields

EXAMPLE

Apache Nutch Node Schema Example

6 fields

EXAMPLE

Apache Nutch Nutch Config Example

3 fields

EXAMPLE

Apache Nutch Nutch Server Info Example

4 fields

EXAMPLE

Apache Nutch Reader Config Example

1 fields

EXAMPLE

Apache Nutch Seed List Example

4 fields

EXAMPLE

Apache Nutch Seed Url Example

2 fields

EXAMPLE

Apache Nutch Service Config Example

3 fields

EXAMPLE

Apache Nutch Service Info Example

1 fields

EXAMPLE

Apache Nutch State Example

0 fields

EXAMPLE

Scroll for all 16

Security Posture 3

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Apache Nutch Authentication

http · 1 scheme

SECURITY

Apache Nutch Domain Security

TLSv1.3 · HSTS · DMARC

SECURITY

Apache Nutch Vulnerability Disclosure

security.txt · contact published

SECURITY

Agentic Access 1

Recommended x-agentic-access execution contracts for AI agents.

Apache Nutch Agentic Access

24 operations · 10 acting

AGENTIC

Use Cases 6

What developers build with this provider.

Enterprise Search

Build enterprise search engines over internal or external web content using Nutch as the crawler and Solr/Elasticsearch as the search backend.

Research Data Collection

Academic and research teams use Nutch for large-scale systematic web data collection and indexing.

Intranet Document Search

Crawl and index intranet sites, wikis, and document repositories for internal enterprise search.

Web Archive Creation

Create structured web archives compatible with CommonCrawl format for long-term data preservation.

SEO and Content Monitoring

Monitor web content changes, track competitor sites, and analyze web structure at scale.

Custom Data Extraction Pipelines

Build custom extraction pipelines using Nutch plugin architecture for targeted data acquisition tasks.

Integrations 5

Pre-built integrations with other platforms and tools.

Apache Solr

Native index writer plugin for indexing crawled content into Apache Solr for full-text search.

Elasticsearch

Index writer plugin for sending crawled content to Elasticsearch clusters.

Apache Hadoop

Core dependency providing distributed storage and processing via HDFS and MapReduce.

Apache Tika

Used for content detection and extraction from a wide range of document formats during parsing.

SolrCloud

Support for SolrCloud distributed search clusters for scalable indexing.

Resources

Get Started 1

Portal, sign-up, and the first successful call

Nutch Tutorial

GettingStarted

Documentation 1

Reference material describing how the API behaves

Apache Nutch Documentation

Documentation

Agent Surfaces 1

MCP servers, agent skills, and machine-readable catalogs

AgenticAccess

AgenticAccess

Design & Contract 3

Pagination, idempotency, versioning, errors, and events

Apache Nutch Spectral Rules

SpectralRules

Apache Nutch Vocabulary

Vocabulary

Apache Nutch JSON-LD Context

JSONLD

Build 3

SDKs, sample code, and the tooling you integrate with

PostmanWorkspace

PostmanWorkspace

Apache Nutch GitHub Repository

GitHubRepository

Apache Software Foundation GitHub

GitHubOrganization

Access & Security 3

Authentication, authorization, and security posture

VulnerabilityDisclosure

VulnerabilityDisclosure

DomainSecurity

DomainSecurity

Authentication

Authentication

Learn 1

Tutorials, courses, talks, and written guidance

Apache Nutch Tutorials

Tutorials

Operate 4

Status, limits, changes, and where to get help

Apache Nutch FAQs

FAQ

Nutch Release Notes

ReleaseNotes

Mailing Lists

Support

Nutch on Stack Overflow

StackOverflow

Commercial 1

Pricing, plans, and the legal terms of use

Apache License 2.0

TermsOfService

Company 1

The organization behind the API

Blog

Blog

Source (apis.yml)

aid: apache-nutch
name: Apache Nutch
description: Apache Nutch is a highly extensible and scalable open-source web crawler software project built on Apache Hadoop
  data structures for batch processing. It provides a pluggable architecture supporting custom parse filters, scoring filters,
  index writers, and protocol implementations. Nutch integrates with Apache Solr and Elasticsearch for full-text search and
  exposes a REST API for managing crawl jobs, configurations, seed lists, and database queries. Governed by the Apache Software
  Foundation under the Apache License 2.0.
type: Index
accessModel:
  pricing: freemium
  onboarding: self-serve
  trial: false
  try_now: true
  public: false
  label: Freemium · Self-serve signup
  confidence: high
  source:
  - plans
  - authentication
  generated: '2026-07-22'
  method: derived
position: Consumer
access: 3rd-Party
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/apache-nutch.png
tags:
- Web Crawler
- Indexing
- Search
- Apache
- Java
- Hadoop
- Open Source
created: '2026-03-16'
modified: '2026-05-19'
url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/apis.yml
specificationVersion: '0.19'
apis:
- aid: apache-nutch:apache-nutch-admin-api
  name: Apache Nutch Admin API
  description: Server administration operations
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Admin
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-admin-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-configuration-api
  name: Apache Nutch Configuration API
  description: Manage Nutch configurations
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Configuration
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-configuration-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-database-api
  name: Apache Nutch Database API
  description: Query the CrawlDB and FetchDB
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Database
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-database-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-job-api
  name: Apache Nutch Job API
  description: Manage crawl jobs
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Job
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-job-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-reader-api
  name: Apache Nutch Reader API
  description: Read sequence files and webgraph data
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Reader
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-reader-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-seed-api
  name: Apache Nutch Seed API
  description: Manage seed URL lists
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Seed
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-seed-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
- aid: apache-nutch:apache-nutch-services-api
  name: Apache Nutch Services API
  description: Auxiliary service operations such as CommonCrawl data dumps
  humanURL: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  baseURL: http://localhost:8081
  tags:
  - Services
  properties:
  - type: OpenAPI
    url: openapi/apache-nutch-services-api-openapi.yml
  - type: Documentation
    url: https://cwiki.apache.org/confluence/display/NUTCH/Nutch+1.X+REST+API
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-config-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-job-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-nutch-server-info-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-seed-list-schema.json
  - type: JSONSchema
    url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-schema/apache-nutch-db-query-schema.json
common:
- type: PostmanWorkspace
  url: https://www.postman.com/kinlaneapi/apache-nutch/overview
- type: AgenticAccess
  url: agentic-access/apache-nutch-agentic-access.yml
- type: VulnerabilityDisclosure
  url: security/apache-nutch-vulnerability-disclosure.yml
- type: DomainSecurity
  url: security/apache-nutch-domain-security.yml
- type: Authentication
  url: authentication/apache-nutch-authentication.yml
- type: GitHubRepository
  url: https://github.com/apache/nutch
  title: Apache Nutch GitHub Repository
- type: GitHubOrganization
  url: https://github.com/apache
  title: Apache Software Foundation GitHub
- type: Documentation
  url: https://nutch.apache.org/documentation/
  title: Apache Nutch Documentation
- type: GettingStarted
  url: https://cwiki.apache.org/confluence/display/NUTCH/NutchTutorial
  title: Nutch Tutorial
- type: Tutorials
  url: https://nutch.apache.org/documentation/tutorials/
  title: Apache Nutch Tutorials
- type: FAQ
  url: https://nutch.apache.org/documentation/faqs/
  title: Apache Nutch FAQs
- type: ReleaseNotes
  url: https://github.com/apache/nutch/blob/master/CHANGES.md
  title: Nutch Release Notes
- type: TermsOfService
  url: https://www.apache.org/licenses/LICENSE-2.0
  title: Apache License 2.0
- type: Support
  url: https://nutch.apache.org/community/mailing-lists/
  title: Mailing Lists
- type: StackOverflow
  url: https://stackoverflow.com/questions/tagged/nutch
  title: Nutch on Stack Overflow
- type: SpectralRules
  url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/rules/apache-nutch-spectral-rules.yml
  title: Apache Nutch Spectral Rules
- type: Vocabulary
  url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/vocabulary/apache-nutch-vocabulary.yaml
  title: Apache Nutch Vocabulary
- type: JSONLD
  url: https://raw.githubusercontent.com/api-evangelist/apache-nutch/refs/heads/main/json-ld/apache-nutch-context.jsonld
  title: Apache Nutch JSON-LD Context
- type: Features
  data:
  - name: Scalable Batch Crawling
    description: Leverages Apache Hadoop data structures for distributed, large-scale web crawling batch processing.
  - name: Pluggable Architecture
    description: Extensible plugin system supporting custom parse filters, scoring filters, index writers, protocol plugins,
      and URL filters.
  - name: REST API for Crawl Management
    description: Full REST API for managing crawl jobs, configurations, seed lists, CrawlDB/FetchDB queries, and sequence
      file readers.
  - name: Full-Text Search Integration
    description: Built-in index writers for Apache Solr and Elasticsearch to enable full-text search over crawled content.
  - name: Apache Tika Parsing
    description: Uses Apache Tika for parsing a wide variety of document formats during the crawl pipeline.
  - name: Duplicate Detection
    description: Built-in deduplication support to identify and remove duplicate content from the crawl database and search
      index.
  - name: Configurable URL Filtering
    description: Regex-based and custom URL filter plugins to control crawl scope and exclusions.
  - name: Incremental Crawling
    description: Supports multi-round incremental crawling workflows to keep the crawl database fresh.
  - name: CommonCrawl Export
    description: Service operations for exporting crawl data in CommonCrawl-compatible formats.
  - name: HTTP Authentication Support
    description: Configurable HTTP authentication schemes for crawling password-protected sites.
- type: UseCases
  data:
  - name: Enterprise Search
    description: Build enterprise search engines over internal or external web content using Nutch as the crawler and Solr/Elasticsearch
      as the search backend.
  - name: Research Data Collection
    description: Academic and research teams use Nutch for large-scale systematic web data collection and indexing.
  - name: Intranet Document Search
    description: Crawl and index intranet sites, wikis, and document repositories for internal enterprise search.
  - name: Web Archive Creation
    description: Create structured web archives compatible with CommonCrawl format for long-term data preservation.
  - name: SEO and Content Monitoring
    description: Monitor web content changes, track competitor sites, and analyze web structure at scale.
  - name: Custom Data Extraction Pipelines
    description: Build custom extraction pipelines using Nutch plugin architecture for targeted data acquisition tasks.
- type: Integrations
  data:
  - name: Apache Solr
    description: Native index writer plugin for indexing crawled content into Apache Solr for full-text search.
  - name: Elasticsearch
    description: Index writer plugin for sending crawled content to Elasticsearch clusters.
  - name: Apache Hadoop
    description: Core dependency providing distributed storage and processing via HDFS and MapReduce.
  - name: Apache Tika
    description: Used for content detection and extraction from a wide range of document formats during parsing.
  - name: SolrCloud
    description: Support for SolrCloud distributed search clusters for scalable indexing.
- url: https://nutch.apache.org/index.xml
  type: Blog
maintainers:
- FN: Kin Lane
  email: info@apievangelist.com