AI Crawler Index

Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 56 crawlers from 30 operators, 1887 IPv4 and 1056 IPv6 prefixes mirrored from 12 operator-published endpoints and refreshed every six hours. Read-only static files: no key, no signup, no rate limit, CORS open, CC0.

AI Crawler Index publishes 1 API on the APIs.io network: AI Crawler Index. Tagged areas include AI crawlers, web crawlers, robots.txt, user agents, and bot detection.

AI Crawler Index’s developer surface includes documentation, support, changelog, and 16 more developer resources.

🌐 Visit website 📡 Source on GitHub

44.2/100 developing ▬ flat Agent 28/100 agent aware Full breakdown ↓
scored 2026-09-02 · rubric v0.18.0

AccessFreeOpen⚡ Free to try

1 APIs 2 MCP Servers

AI crawlersweb crawlersrobots.txtuser agentsbot detectionGPTBotClaudeBotcrawler IP rangesllms.txtopen data

Kin Score

Kin Score How this is scored →
scored 2026-09-02 · rubric v0.18.0

Composite quality — 44.2/100 · developing

Contract Quality 8.8 / 21

Developer Ergonomics 9.6 / 17

Access Clarity 3.6 / 17

Operational Transparency 4.7 / 11

Contract Governance 1.9 / 10

Discoverability 6.5 / 9

Regulatory Posture 6.4 / 15

Agent readiness — 28/100 · agent aware

Machine-Readable Contract 18 / 18

Agentic Access Contract 0 / 10

Documented Reversibility 0 / 6

MCP Server 12 / 12

Machine-Readable Auth 10 / 10

Idempotency 0 / 9

Stable Error Semantics 8 / 8

Request/Response Examples 0 / 7

Rate-Limit Signaling 7 / 7

Typed Event Surface 0 / 6

Agent Skills 5 / 5

Well-Known Catalog 4 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Delegated User Identity 0 / 6

Protected Resource Metadata 0 / 5

Registration Without a Human 0 / 6

Agentic Commerce Surface 0 / 5

Regulatory Posture applies to this provider. Its tags matched the Government & Public Sector regime, so Regulatory Posture carries 15 points of the composite. If this regime is wrong for your business, say so on your provider repo — the applicability map is public and we will correct it.

The six quality facets above are damped to 85 points between them, because the conditional facet above carries the other 15. That is why each facet's contribution is shown against a damped maximum: raising a quality facet moves the composite by 85% of its nominal weight, not 100%. The full arithmetic is at apis.io/rating/.

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from github.com/api-evangelist/pathwren: open an issue to ask a question, or submit a pull request to add artifacts. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

APIs 1

Individual APIs this provider publishes, each with its own machine-readable definition.

AI Crawler Index

Which crawler a User-Agent header belongs to, who operates it, whether it documents obeying robots.txt, how to verify it is genuine against the operator's own published IP range...

MCP Servers 2

Model Context Protocol servers that expose these APIs to AI agents.

Rate Limits 1

Documented rate limits and quota policies.

Pathwren Rate Limits

0 limits

RATE LIMITS

JSON Schema 1

Standalone JSON Schema definitions for this provider's data models.

Get Started 1

Portal, sign-up, and the first successful call

DeveloperPortal

DeveloperPortal

Documentation 1

Reference material describing how the API behaves

Documentation

Documentation

Agent Surfaces 4

MCP servers, agent skills, and machine-readable catalogs

WellKnown

LLMsTxt

MCPServer

AgentSkill

Design & Contract 2

Pagination, idempotency, versioning, errors, and events

Conformance

Conformance

Lifecycle

Lifecycle

Build 1

SDKs, sample code, and the tooling you integrate with

Packages

Packages

Access & Security 4

Authentication, authorization, and security posture

SecurityTxt

SecurityTxt

Security

Security

VulnerabilityDisclosure

VulnerabilityDisclosure

DomainSecurity

DomainSecurity

Operate 3

Status, limits, changes, and where to get help

Support

Support

StatusPage

StatusPage

ChangeLog

ChangeLog

Commercial 2

Pricing, plans, and the legal terms of use

License

License

Plans

Plans

Other 1

Properties that don't map to a standard resource type

AgentCard

AgentCard

Source (apis.yml)

aid: pathwren
name: AI Crawler Index
type: Index
description: 'Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes
  — as JSON, CSV, robots.txt and regex. 56 crawlers from 30 operators, 1887 IPv4 and 1056 IPv6 prefixes mirrored from 12 operator-published
  endpoints and refreshed every six hours. Read-only static files: no key, no signup, no rate limit, CORS open, CC0.'
accessModel:
  pricing: free
  onboarding: open
  trial: false
  try_now: true
  public: true
  label: Free · Open access
  confidence: high
  source:
  - plans
  - authentication
  - rate-limits
  - security
  generated: '2026-09-02'
  method: derived
image: https://www.pathwren.workers.dev/icon.png
url: https://www.pathwren.workers.dev/apis.json
specificationVersion: '1.0'
created: '2026-08-31'
modified: '2026-09-01'
tags:
- AI crawlers
- web crawlers
- robots.txt
- user agents
- bot detection
- GPTBot
- ClaudeBot
- crawler IP ranges
- llms.txt
- open data
maintainers:
- FN: Pathwren
  email: pathwren@tutamail.com
  url: https://www.pathwren.workers.dev/about.html
apis:
- aid: www.pathwren.workers.dev:ai-crawler-index
  name: AI Crawler Index
  description: 'Which crawler a User-Agent header belongs to, who operates it, whether it documents obeying robots.txt, how
    to verify it is genuine against the operator''s own published IP ranges, and what blocking it costs you. Every endpoint
    is a GET returning JSON, CSV or plain text.


    HOST NOTE: the entire surface is served from www.pathwren.workers.dev, a Cloudflare Workers default subdomain rather than
    a custom domain. That is NOT an ephemeral tunnel -- unlike a trycloudflare quick tunnel, a workers.dev subdomain is stable
    and tied to the account -- but it is a platform-owned address, so the operator does not control it and it cannot be moved.
    Recorded rather than treated as disqualifying: everything it serves is real and complete. Verified 2026-09-01.'
  image: https://www.pathwren.workers.dev/icon.png
  humanURL: https://www.pathwren.workers.dev/api.html?s=ai-crawler-index&c=apis-json
  baseURL: https://www.pathwren.workers.dev
  tags:
  - AI crawlers
  - web crawlers
  - robots.txt
  - user agents
  - bot detection
  - GPTBot
  - ClaudeBot
  - crawler IP ranges
  - llms.txt
  - open data
  properties:
  - type: OpenAPI
    url: https://www.pathwren.workers.dev/openapi.json
  - type: x-openapi-yaml
    url: https://www.pathwren.workers.dev/openapi.yaml
  - type: Swagger
    url: https://www.pathwren.workers.dev/swagger.json
  - type: Documentation
    url: https://www.pathwren.workers.dev/api.html
  - type: Feed
    url: https://www.pathwren.workers.dev/feed.xml
  - type: Feed
    url: https://www.pathwren.workers.dev/feed.json
  - type: x-mcp-server
    url: https://www.pathwren.workers.dev/mcp
  - type: x-mcp-manifest
    url: https://www.pathwren.workers.dev/.well-known/mcp.json
  - type: x-onboarding
    url: https://www.pathwren.workers.dev/.well-known/api-onboarding
  - type: x-plugin-manifest
    url: https://www.pathwren.workers.dev/.well-known/ai-plugin.json
  - type: x-api-catalog
    url: https://www.pathwren.workers.dev/.well-known/api-catalog
  - type: x-llms-txt
    url: https://www.pathwren.workers.dev/llms.txt
  - type: x-bulk-data
    url: https://www.pathwren.workers.dev/data/agents.json
  - type: x-ip-ranges
    url: https://www.pathwren.workers.dev/ip-ranges/all.json
  - type: x-security-txt
    url: https://www.pathwren.workers.dev/.well-known/security.txt
  - type: x-sitemap
    url: https://www.pathwren.workers.dev/sitemap.xml
  - type: StatusPage
    url: https://www.pathwren.workers.dev/status.html
  - type: License
    url: https://creativecommons.org/publicdomain/zero/1.0/
  - type: MCPServer
    url: https://www.pathwren.workers.dev/mcp
    description: Mirrored from the provider's own x-mcp-server property so the canonical reader sees it. Verified 2026-09-01.
  - type: OpenAPI
    url: openapi/pathwren-ai-crawler-index-openapi.json
    description: Verbatim copy of the provider's OpenAPI 3.1 document, fetched 2026-09-01 (HTTP 200).
  - type: Overlay
    url: overlays/pathwren-ai-crawler-index-overlay.yaml
  - type: JSONSchema
    url: json-schema/pathwren-crawler.schema.json
  - type: Authentication
    url: authentication/pathwren-authentication.yml
  - type: Conventions
    url: conventions/pathwren-conventions.yml
  - type: ErrorCatalog
    url: errors/pathwren-problem-types.yml
  - type: RateLimits
    url: rate-limits/pathwren-rate-limits.yml
  - type: DataModel
    url: data-model/pathwren-data-model.yml
  - type: ToolCrosswalk
    url: mcp/pathwren-tool-crosswalk.yml
  - type: GettingStarted
    url: https://www.pathwren.workers.dev/.well-known/api-onboarding
  - type: APIReference
    url: https://www.pathwren.workers.dev/api.html
  - type: ChangeLog
    url: https://www.pathwren.workers.dev/changelog.html
  contact:
  - FN: Pathwren
    email: pathwren@tutamail.com
    url: https://www.pathwren.workers.dev/about.html
common:
- type: DeveloperPortal
  url: https://www.pathwren.workers.dev/
- type: Documentation
  url: https://www.pathwren.workers.dev/api.html
- type: Support
  url: https://www.pathwren.workers.dev/about.html
- type: License
  url: https://creativecommons.org/publicdomain/zero/1.0/
- type: StatusPage
  url: https://www.pathwren.workers.dev/status.html
- type: ChangeLog
  url: changelog/pathwren-changelog.yml
- type: WellKnown
  url: well-known/pathwren-well-known.yml
- type: SecurityTxt
  url: well-known/pathwren-security.txt
- type: Security
  url: security/pathwren-vulnerability-disclosure.yml
- type: VulnerabilityDisclosure
  url: security/pathwren-vulnerability-disclosure.yml
- type: DomainSecurity
  url: security/pathwren-domain-security.yml
- type: LLMsTxt
  url: llms/pathwren-llms.txt
- type: MCPServer
  url: mcp/pathwren-mcp.yml
- type: AgentCard
  url: a2a/pathwren-a2a.yml
- type: AgentSkill
  url: skills/_index.yml
- type: Packages
  url: packages/pathwren-packages.yml
- type: Conformance
  url: conformance/pathwren-conformance.yml
- type: Lifecycle
  url: lifecycle/pathwren-lifecycle.yml
- type: Plans
  url: plans/pathwren-plans-pricing.yml
x-enrichment:
  date: '2026-09-01'
  status: enriched
  artifacts_added: 38
  pass: local-v1

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this

find_providersBrowse and filter every provider in the catalog.
get_provider_artifactsEvery artifact this provider publishes, grouped by type.
get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
get_provider_ratingPRO — composite, band, trend and facet scores.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This provider

curl "https://apis.io/api/v1/providers/pathwren"

All providers

curl "https://apis.io/api/v1/providers?limit=25"

Every operation they expose

curl "https://apis.io/api/v1/providers/pathwren/operations?limit=25"

How their score was established

curl "https://apis.io/api/v1/providers/pathwren/evidence"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no email required.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

AI Crawler Index

Kin Score

APIs 1

AI Crawler Index

MCP Servers 2

AI Crawler Index MCP Server

AI Crawler Index MCP Server

Pricing Plans 1

Pathwren Plans Pricing

Rate Limits 1

Pathwren Rate Limits

JSON Schema 1

Crawler

Security Posture 3

Pathwren Authentication

Pathwren Domain Security

Pathwren Vulnerability Disclosure

Resources

Get Started 1

Documentation 1

Agent Surfaces 4

Design & Contract 2

Build 1

Access & Security 4

Operate 3

Commercial 2

Other 1

Source (apis.yml)

Work with this as data

MCP server

Tools for providers

Call it yourself

Get an API key