# AI Crawler Index

**Canonical:** https://apis.io/providers/pathwren/  
**Website:** https://www.pathwren.workers.dev/  
**APIs profiled:** 1

Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 56 crawlers from 30 operators, 1887 IPv4 and 1056 IPv6 prefixes mirrored from 12 operator-published endpoints and refreshed every six hours. Read-only static files: no key, no signup, no rate limit, CORS open, CC0.

## Kin Score — 44.2 / 100 (developing)

Scored 2026-09-02 under rubric 0.18.0. Trend: flat (+0.0 from 44.2).

| Facet | Score |
|---|---|
| Discoverability | 75.9 |
| Contract Quality | 41.3 |
| Governance | 18.2 |
| Contract Governance | 18.2 |
| Operational Transparency | 42.1 |
| Developer Ergonomics | 56.5 |
| Commercial Clarity | 21.1 |
| Access Clarity | 21.1 |

Regulatory layer — **Government & Public Sector**: 42.6 (matched via tags).

## Agent readiness — 28.1 (agent-aware)

| Dimension | Value |
|---|---|
| Spec Presence | yes |
| Agentic Access | no |
| Reversibility Documented | no |
| MCP Server | documented |
| Auth Clarity | bearer |
| Idempotency | no |
| Error Semantics | documented |
| OpenAPI Examples | no |
| Rate Limit Signal | documented |
| Event Surface Described | no |
| Agent Skills | derived |
| Well Known Catalog | yes |
| Consent Identity | no |
| Agent Card | no |
| Dry Run Mode | no |
| Delegated Identity | no |
| Protected Resource Metadata | no |
| Dynamic Client Registration | no |
| Agentic Commerce | no |

## Access

Free · Open access — onboarding: open, pricing: free, trial: no (confidence: high).

## APIs (1)

- **AI Crawler Index** — Which crawler a User-Agent header belongs to, who operates it, whether it documents obeying robots.txt, how to verify it is genuine against the operator's own published IP range...

## MCP servers (2)

- **AI Crawler Index MCP Server** — Mirrored from the provider's own x-mcp-server property so the canonical reader sees it. Verified 2026-09-01.
- **AI Crawler Index MCP Server**

## Security (3)

- **Pathwren Authentication** — 0 schemes
- **Pathwren Domain Security** — TLSv1.3 · HSTS · DMARC
- **Pathwren Vulnerability Disclosure** — security.txt · contact published

## Plans (1)

- **Pathwren Plans Pricing**

## Tags

AI crawlers, web crawlers, robots.txt, user agents, bot detection, GPTBot, ClaudeBot, crawler IP ranges, llms.txt, open data

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/providers/pathwren/). Scores are computed from the provider's own public artifacts under a published rubric.