# AI Crawler Index Crawlers API

**Canonical:** https://apis.io/apis/pathwren/pathwren-crawlers-api/  
**Provider:** AI Crawler Index — https://apis.io/providers/pathwren/  
**Base URL:** https://www.pathwren.workers.dev  
**Documentation:** https://www.pathwren.workers.dev/api.html?s=ai-crawler-index&c=apis-json

AI Crawler Index Crawlers API is one of 5 APIs that [AI Crawler Index](https://apis.io/providers/pathwren/) publishes on the [APIs.io](https://apis.io/) network, described by a machine-readable OpenAPI specification. This API exposes 1 JSON Schema definition. Tagged areas include Crawlers. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, authentication docs, rate-limit docs, a getting-started guide, an API reference, and 1 JSON Schema.

One record per crawler.

## Operations (57)

| Method | Path | Summary |
|---|---|---|
| GET | `/crawler/{slug}.json` | One crawler record by slug |
| GET | `/crawler/gptbot.json` | Record for GPTBot |
| GET | `/crawler/oai-searchbot.json` | Record for OAI-SearchBot |
| GET | `/crawler/chatgpt-user.json` | Record for ChatGPT-User |
| GET | `/crawler/claudebot.json` | Record for ClaudeBot |
| GET | `/crawler/claude-searchbot.json` | Record for Claude-SearchBot |
| GET | `/crawler/claude-user.json` | Record for Claude-User |
| GET | `/crawler/anthropic-ai.json` | Record for anthropic-ai |
| GET | `/crawler/claude-web.json` | Record for Claude-Web |
| GET | `/crawler/google-extended.json` | Record for Google-Extended |
| GET | `/crawler/googlebot.json` | Record for Googlebot |
| GET | `/crawler/googleother.json` | Record for GoogleOther |
| GET | `/crawler/google-cloudvertexbot.json` | Record for Google-CloudVertexBot |
| GET | `/crawler/google-inspectiontool.json` | Record for Google-InspectionTool |
| GET | `/crawler/googlebot-image.json` | Record for Googlebot-Image |
| GET | `/crawler/googlebot-news.json` | Record for Googlebot-News |
| GET | `/crawler/storebot-google.json` | Record for Storebot-Google |
| GET | `/crawler/bingbot.json` | Record for bingbot |
| GET | `/crawler/applebot.json` | Record for Applebot |
| GET | `/crawler/applebot-extended.json` | Record for Applebot-Extended |
| GET | `/crawler/perplexitybot.json` | Record for PerplexityBot |
| GET | `/crawler/perplexity-user.json` | Record for Perplexity-User |
| GET | `/crawler/ccbot.json` | Record for CCBot |
| GET | `/crawler/bytespider.json` | Record for Bytespider |
| GET | `/crawler/tiktokspider.json` | Record for TikTokSpider |
| GET | `/crawler/meta-externalagent.json` | Record for meta-externalagent |
| GET | `/crawler/meta-externalfetcher.json` | Record for meta-externalfetcher |
| GET | `/crawler/facebookexternalhit.json` | Record for facebookexternalhit |
| GET | `/crawler/facebookbot.json` | Record for FacebookBot |
| GET | `/crawler/amazonbot.json` | Record for Amazonbot |
| GET | `/crawler/duckassistbot.json` | Record for DuckAssistBot |
| GET | `/crawler/duckduckbot.json` | Record for DuckDuckBot |
| GET | `/crawler/ai2bot.json` | Record for AI2Bot |
| GET | `/crawler/ai2bot-dolma.json` | Record for Ai2Bot-Dolma |
| GET | `/crawler/cohere-ai.json` | Record for cohere-ai |
| GET | `/crawler/cohere-training-data-crawler.json` | Record for cohere-training-data-crawler |
| GET | `/crawler/mistralai-user.json` | Record for MistralAI-User |
| GET | `/crawler/youbot.json` | Record for YouBot |
| GET | `/crawler/diffbot.json` | Record for Diffbot |
| GET | `/crawler/omgilibot.json` | Record for omgilibot |
| GET | `/crawler/omgili.json` | Record for omgili |
| GET | `/crawler/webzio-extended.json` | Record for Webzio-Extended |
| GET | `/crawler/imagesiftbot.json` | Record for ImagesiftBot |
| GET | `/crawler/timpibot.json` | Record for Timpibot |
| GET | `/crawler/semrushbot.json` | Record for SemrushBot |
| GET | `/crawler/semrushbot-ocob.json` | Record for SemrushBot-OCOB |
| GET | `/crawler/ahrefsbot.json` | Record for AhrefsBot |
| GET | `/crawler/archive-org-bot.json` | Record for archive.org_bot |
| GET | `/crawler/ia-archiver.json` | Record for ia_archiver |
| GET | `/crawler/yandexbot.json` | Record for YandexBot |
| GET | `/crawler/baiduspider.json` | Record for Baiduspider |
| GET | `/crawler/seznambot.json` | Record for SeznamBot |
| GET | `/crawler/yeti.json` | Record for Yeti |
| GET | `/crawler/petalbot.json` | Record for PetalBot |
| GET | `/crawler/firecrawlagent.json` | Record for FirecrawlAgent |
| GET | `/crawler/scrapy.json` | Record for Scrapy |
| GET | `/crawler/img2dataset.json` | Record for img2dataset |

## Machine-readable artifacts (28)

- **OpenAPI** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/openapi/pathwren-crawlers-api-openapi.yml
- **x-openapi-yaml** — https://www.pathwren.workers.dev/openapi.yaml
- **Swagger** — https://www.pathwren.workers.dev/swagger.json
- **Documentation** — https://www.pathwren.workers.dev/api.html
- **Feed** — https://www.pathwren.workers.dev/feed.xml
- **Feed** — https://www.pathwren.workers.dev/feed.json
- **x-mcp-server** — https://www.pathwren.workers.dev/mcp
- **x-mcp-manifest** — https://www.pathwren.workers.dev/.well-known/mcp.json
- **x-onboarding** — https://www.pathwren.workers.dev/.well-known/api-onboarding
- **x-plugin-manifest** — https://www.pathwren.workers.dev/.well-known/ai-plugin.json
- **x-api-catalog** — https://www.pathwren.workers.dev/.well-known/api-catalog
- **x-llms-txt** — https://www.pathwren.workers.dev/llms.txt
- **x-bulk-data** — https://www.pathwren.workers.dev/data/agents.json
- **x-ip-ranges** — https://www.pathwren.workers.dev/ip-ranges/all.json
- **x-security-txt** — https://www.pathwren.workers.dev/.well-known/security.txt
- **x-sitemap** — https://www.pathwren.workers.dev/sitemap.xml
- **StatusPage** — https://www.pathwren.workers.dev/status.html
- **License** — https://creativecommons.org/publicdomain/zero/1.0/
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/json-schema/pathwren-crawler.schema.json
- **Authentication** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/authentication/pathwren-authentication.yml
- **Conventions** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/conventions/pathwren-conventions.yml
- **ErrorCatalog** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/errors/pathwren-problem-types.yml
- **RateLimits** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/rate-limits/pathwren-rate-limits.yml
- **DataModel** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/data-model/pathwren-data-model.yml
- **ToolCrosswalk** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/mcp/pathwren-tool-crosswalk.yml
- **GettingStarted** — https://www.pathwren.workers.dev/.well-known/api-onboarding
- **APIReference** — https://www.pathwren.workers.dev/api.html
- **ChangeLog** — https://www.pathwren.workers.dev/changelog.html

## Other AI Crawler Index APIs (4)

- [AI Crawler Index Bulk API](https://apis.io/apis/pathwren/pathwren-bulk-api/)
- [AI Crawler Index Ip Ranges API](https://apis.io/apis/pathwren/pathwren-ip-ranges-api/)
- [AI Crawler Index Robots API](https://apis.io/apis/pathwren/pathwren-robots-api/)
- [AI Crawler Index Status API](https://apis.io/apis/pathwren/pathwren-status-api/)

## Tags

Crawlers

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/apis/pathwren/pathwren-crawlers-api/). The API's provider profile, Kin Score and agent-readiness rating are at https://apis.io/providers/pathwren/.