AI Crawler Index
Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Agent, look one up, list them by category or operator, generate a robots.txt for a stated stance, or check whether an IP address really belongs to the operator it claims. Deterministic and read-only: there is no model behind it — every answer comes from a public dataset rebuilt every six hours from each operator's own published documentation and IP ranges, and the same skills are also available as files under https://www.pathwren.workers.dev/ and as MCP tools at https://www.pathwren.workers.dev/mcp. No key, no signup, no quota. Independent and unaffiliated with any operator it documents.
AgentCard object — a capabilities object, a declared protocolVersion, and a well-formed skills array.
An A2A client finds this agent by fetching the well-known path on the provider's own host. That is what makes an agent card different from every other agent artifact in this catalog: it is provider-published by construction — it cannot be derived, generated, or reconstructed on a provider's behalf.
https://www.pathwren.workers.dev/a2a
· transport JSONRPC
non-standard supportedInterfaces
Classify a user-agent string classify_user_agent
Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training.
user-agentcrawlersbot detectionidentificationLook up one crawler lookup_crawler
The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'.
crawlersreferencerobots.txtlookupList and filter crawlers list_crawlers
The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.
crawlerscataloguefilterdiscoveryGenerate a robots.txt for a stance generate_robots_txt
A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.
robots.txtpolicyconfigurationgeneratorCheck an IP against operator-published crawler ranges is_verified_crawler_ip
Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'.
ip rangesverificationbot detectionsecurityWhat changed in the index whats_changed
Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'.
freshnesschangelogmonitoringdata quality2026-09-01 from https://www.pathwren.workers.dev/.well-known/agent-card.json,
HTTP 200. The body was saved verbatim and is the sole source for
everything on this page — no field is inferred.
View the captured card.
Providers change what they serve; if this card has moved or changed shape,
the provider profile carries the
current state as of the last build.
Work with this as data
Every agent card here is available over the APIs.io API and to AI agents over MCP. A2A Agent Cards is not yet its own endpoint on the v1 API. Reach it through catalog search and the tag graph, or the MCP server.