# AI Crawler Index Robots API

**Canonical:** https://apis.io/apis/pathwren/pathwren-robots-api/  
**Provider:** AI Crawler Index — https://apis.io/providers/pathwren/  
**Base URL:** https://www.pathwren.workers.dev  
**Documentation:** https://www.pathwren.workers.dev/api.html?s=ai-crawler-index&c=apis-json

AI Crawler Index Robots API is one of 5 APIs that [AI Crawler Index](https://apis.io/providers/pathwren/) publishes on the [APIs.io](https://apis.io/) network, described by a machine-readable OpenAPI specification. This API exposes 1 JSON Schema definition. Tagged areas include Robots. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, authentication docs, rate-limit docs, a getting-started guide, an API reference, and 1 JSON Schema.

Ready-made robots.txt policy files.

## Operations (2)

| Method | Path | Summary |
|---|---|---|
| GET | `/robots/{policy}.txt` | A ready-made robots.txt policy file |
| GET | `/policy/{policy}.json` | A policy with its rationale and the crawlers it names |

## Machine-readable artifacts (28)

- **OpenAPI** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/openapi/pathwren-robots-api-openapi.yml
- **x-openapi-yaml** — https://www.pathwren.workers.dev/openapi.yaml
- **Swagger** — https://www.pathwren.workers.dev/swagger.json
- **Documentation** — https://www.pathwren.workers.dev/api.html
- **Feed** — https://www.pathwren.workers.dev/feed.xml
- **Feed** — https://www.pathwren.workers.dev/feed.json
- **x-mcp-server** — https://www.pathwren.workers.dev/mcp
- **x-mcp-manifest** — https://www.pathwren.workers.dev/.well-known/mcp.json
- **x-onboarding** — https://www.pathwren.workers.dev/.well-known/api-onboarding
- **x-plugin-manifest** — https://www.pathwren.workers.dev/.well-known/ai-plugin.json
- **x-api-catalog** — https://www.pathwren.workers.dev/.well-known/api-catalog
- **x-llms-txt** — https://www.pathwren.workers.dev/llms.txt
- **x-bulk-data** — https://www.pathwren.workers.dev/data/agents.json
- **x-ip-ranges** — https://www.pathwren.workers.dev/ip-ranges/all.json
- **x-security-txt** — https://www.pathwren.workers.dev/.well-known/security.txt
- **x-sitemap** — https://www.pathwren.workers.dev/sitemap.xml
- **StatusPage** — https://www.pathwren.workers.dev/status.html
- **License** — https://creativecommons.org/publicdomain/zero/1.0/
- **JSONSchema** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/json-schema/pathwren-crawler.schema.json
- **Authentication** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/authentication/pathwren-authentication.yml
- **Conventions** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/conventions/pathwren-conventions.yml
- **ErrorCatalog** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/errors/pathwren-problem-types.yml
- **RateLimits** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/rate-limits/pathwren-rate-limits.yml
- **DataModel** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/data-model/pathwren-data-model.yml
- **ToolCrosswalk** — https://raw.githubusercontent.com/api-evangelist/pathwren/refs/heads/main/mcp/pathwren-tool-crosswalk.yml
- **GettingStarted** — https://www.pathwren.workers.dev/.well-known/api-onboarding
- **APIReference** — https://www.pathwren.workers.dev/api.html
- **ChangeLog** — https://www.pathwren.workers.dev/changelog.html

## Other AI Crawler Index APIs (4)

- [AI Crawler Index Bulk API](https://apis.io/apis/pathwren/pathwren-bulk-api/)
- [AI Crawler Index Crawlers API](https://apis.io/apis/pathwren/pathwren-crawlers-api/)
- [AI Crawler Index Ip Ranges API](https://apis.io/apis/pathwren/pathwren-ip-ranges-api/)
- [AI Crawler Index Status API](https://apis.io/apis/pathwren/pathwren-status-api/)

## Tags

Robots

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/apis/pathwren/pathwren-robots-api/). The API's provider profile, Kin Score and agent-readiness rating are at https://apis.io/providers/pathwren/.