AI Crawler Index

Which crawler a User-Agent header belongs to, who operates it, whether it documents obeying robots.txt, how to verify it is genuine against the operator's own published IP ranges, and what blocking it costs you. Every endpoint is a GET returning JSON, CSV or plain text. HOST NOTE: the entire surface is served from www.pathwren.workers.dev, a Cloudflare Workers default subdomain rather than a custom domain. That is NOT an ephemeral tunnel -- unlike a trycloudflare quick tunnel, a workers.dev subdomain is stable and tied to the account -- but it is a platform-owned address, so the operator does not control it and it cannot be moved. Recorded rather than treated as disqualifying: everything it serves is real and complete. Verified 2026-09-01.