# AI Crawler Index

**Canonical:** https://apis.io/apis/pathwren/ai-crawler-index/  
**Provider:** AI Crawler Index — https://apis.io/providers/pathwren/  
**Base URL:** https://www.pathwren.workers.dev  
**Documentation:** https://www.pathwren.workers.dev/api.html?s=ai-crawler-index&c=apis-json

AI Crawler Index is published by [AI Crawler Index](https://apis.io/providers/pathwren/) on the [APIs.io](https://apis.io/) network, described by a machine-readable OpenAPI specification. This API exposes 1 JSON Schema definition. Tagged areas include AI crawlers, web crawlers, robots.txt, user agents, and bot detection. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, authentication docs, rate-limit docs, a getting-started guide, an API reference, and 1 JSON Schema.

Which crawler a User-Agent header belongs to, who operates it, whether it documents obeying robots.txt, how to verify it is genuine against the operator's own published IP ranges, and what blocking it costs you. Every endpoint is a GET returning JSON, CSV or plain text. HOST NOTE: the entire surface is served from www.pathwren.workers.dev, a Cloudflare Workers default subdomain rather than a custom domain. That is NOT an ephemeral tunnel -- unlike a trycloudflare quick tunnel, a workers.dev subdomain is stable and tied to the account -- but it is a platform-owned address, so the operator does not control it and it cannot be moved. Recorded rather than treated as disqualifying: everything it serves is real and complete. Verified 2026-09-01.

## Machine-readable artifacts (32)

- **OpenAPI** — https://www.pathwren.workers.dev/openapi.json
- **x-openapi-yaml** — https://www.pathwren.workers.dev/openapi.yaml
- **Swagger** — https://www.pathwren.workers.dev/swagger.json
- **Documentation** — https://www.pathwren.workers.dev/api.html
- **Feed** — https://www.pathwren.workers.dev/feed.xml
- **Feed** — https://www.pathwren.workers.dev/feed.json
- **x-mcp-server** — https://www.pathwren.workers.dev/mcp
- **x-mcp-manifest** — https://www.pathwren.workers.dev/.well-known/mcp.json
- **x-onboarding** — https://www.pathwren.workers.dev/.well-known/api-onboarding
- **x-plugin-manifest** — https://www.pathwren.workers.dev/.well-known/ai-plugin.json
- **x-api-catalog** — https://www.pathwren.workers.dev/.well-known/api-catalog
- **x-llms-txt** — https://www.pathwren.workers.dev/llms.txt
- **x-bulk-data** — https://www.pathwren.workers.dev/data/agents.json
- **x-ip-ranges** — https://www.pathwren.workers.dev/ip-ranges/all.json
- **x-security-txt** — https://www.pathwren.workers.dev/.well-known/security.txt
- **x-sitemap** — https://www.pathwren.workers.dev/sitemap.xml
- **StatusPage** — https://www.pathwren.workers.dev/status.html
- **License** — https://creativecommons.org/publicdomain/zero/1.0/
- **MCPServer** — https://www.pathwren.workers.dev/mcp
- **OpenAPI** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/openapi/pathwren-ai-crawler-index-openapi.json
- **Overlay** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/overlays/pathwren-ai-crawler-index-overlay.yaml
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/json-schema/pathwren-crawler.schema.json
- **Authentication** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/authentication/pathwren-authentication.yml
- **Conventions** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/conventions/pathwren-conventions.yml
- **ErrorCatalog** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/errors/pathwren-problem-types.yml
- **RateLimits** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/rate-limits/pathwren-rate-limits.yml
- **DataModel** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/data-model/pathwren-data-model.yml
- **ToolCrosswalk** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/mcp/pathwren-tool-crosswalk.yml
- **GettingStarted** — https://www.pathwren.workers.dev/.well-known/api-onboarding
- **APIReference** — https://www.pathwren.workers.dev/api.html
- **ChangeLog** — https://www.pathwren.workers.dev/changelog.html
- **APIsJSON** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/apis.yml

## Tags

AI crawlers, web crawlers, robots.txt, user agents, bot detection, GPTBot, ClaudeBot, crawler IP ranges, llms.txt, open data

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/apis/pathwren/ai-crawler-index/). The API's provider profile, Kin Score and agent-readiness rating are at https://apis.io/providers/pathwren/.