AI Crawler Index
Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 56 crawlers from 30 operators, 1887 IPv4 and 1056 IPv6 prefixes mirrored from 12 operator-published endpoints and refreshed every six hours. Read-only static files: no key, no signup, no rate limit, CORS open, CC0.
AI Crawler Index publishes 5 API contracts indexed on the APIs.io network, including Bulk API, Crawlers API, Ip Ranges API, and 2 more. Tagged areas include AI crawlers, web crawlers, Robots.txt, User Agents, and Bot Detection.
AI Crawler Index’s developer surface includes pricing, documentation, support, changelog, and 19 more developer resources.
Kin Score
Standards implemented 1
Standards this provider implements or carries. De facto ones are profiled and measured by the API Commons; formal ones come from the standards directory and are linked by the provider's own tags. The evidence column says how each claim was established — not that it was made.
APIs 5
Individual APIs this provider publishes, each with its own machine-readable definition.
AI Crawler Index Bulk API
The whole dataset in several shapes.
AI Crawler Index Crawlers API
One record per crawler.
AI Crawler Index Ip Ranges API
Operator-published prefixes, normalised.
AI Crawler Index Robots API
Ready-made robots.txt policy files.
AI Crawler Index Status API
Freshness of the upstream sources.
MCP Servers 1
Model Context Protocol servers that expose these APIs to AI agents.
AI Crawler Index MCP Server
Remote MCP server at www.pathwren.workers.dev over streamable HTTP; 9 tools listed.
MCP SERVERPricing Plans 1
Published pricing tiers and plan structures.
Rate Limits 1
Documented rate limits and quota policies.
Pathwren Rate Limits
RATE LIMITSOpenAPI Overlays 1
Overlays applied on top of this provider's contracts. Each card says who wrote it: a document the provider publishes, or one API Evangelist derived or generated.
Pathwren Ai Crawler Index Overlay
GENERATEDJSON Schema 1
Standalone JSON Schema definitions for this provider's data models.
Crawler
JSON SCHEMASecurity Posture 3
Authentication, domain security, vulnerability disclosure, and trust-center signals.
Resources
Get Started 1
Portal, sign-up, and the first successful call
Documentation 1
Reference material describing how the API behaves
Agent Surfaces 4
MCP servers, agent skills, and machine-readable catalogs
Design & Contract 2
Pagination, idempotency, versioning, errors, and events
Build 1
SDKs, sample code, and the tooling you integrate with
Access & Security 4
Authentication, authorization, and security posture
Operate 4
Status, limits, changes, and where to get help
Commercial 3
Pricing, plans, and the legal terms of use
Company 1
The organization behind the API
Other 2
Properties that don't map to a standard resource type
Source (apis.yml)
Work with this as data
Every provider here is available over the APIs.io API and to AI agents over MCP.