Crawl Census

aid: crawlcensus-com
name: Crawl Census
description: Crawl Census (crawlcensus.com) is an independent, unfunded measurement service that audits what AI crawlers and
  answer engines can actually read on any website. It checks robots.txt policy against 39 tracked AI crawler tokens, sends
  live requests under real crawler user agents to catch edge blocking, scores JavaScript-free readability, structured data
  and llms.txt, and publishes a public, CC BY 4.0 record of roughly 30,000 domains and who blocks whom. Access is free via
  the web, a JSON REST API at /api/v1, and a remote MCP server.
url: https://artifacts.apis.io/crawlcensus-com/apis.yml
x-type: company
x-source: harvest:mcp-registry
x-tier: profiled
x-tier-reason: enrichment pass 2026-11-04 (modular-v1; was harvest)
accessModel:
  pricing: free
  onboarding: self-serve
  trial: false
  try_now: true
  public: false
  gated: none
  gated_source:
  - onboarding
  label: Free · Self-serve signup
  confidence: high
  probe_state: covered
  source:
  - plans
  - authentication
  - rate-limits
  - security
  generated: '2026-11-13'
  method: derived
specificationVersion: '0.20'
created: '2026-11-04'
modified: '2026-11-04'
tags:
- Company
- AI Crawlers
- Robots.txt
- SEO
- Web Measurement
- Open Data
- Llms.txt
- MCP
tags_raw:
- Company
- AI Crawlers
- Robots.txt
- SEO
- Web Measurement
- Open Data
- llms.txt
- MCP
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com
- FN: APIs.json
  email: info@apis.io
image: https://crawlcensus.com/og/default.png
common:
- type: RateLimits
  url: rate-limits/crawlcensus-com-rate-limits.yml
- type: Plans
  url: plans/crawlcensus-com-plans-pricing.yml
- type: Conventions
  url: conventions/crawlcensus-com-conventions.yml
- type: Authentication
  url: authentication/crawlcensus-com-authentication.yml
- type: Lifecycle
  url: lifecycle/crawlcensus-com-lifecycle.yml
- type: ErrorCatalog
  url: errors/crawlcensus-com-problem-types.yml
- type: LLMsTxt
  url: llms/crawlcensus-com-llms.txt
- type: MCPServer
  url: mcp/crawlcensus-com-mcp.yml
- type: Hosts
  url: hosts/crawlcensus-com-hosts.yml
- type: Vendors
  url: vendors/crawlcensus-com-vendors.yml
- type: SDKs
  url: packages/crawlcensus-com-packages.yml
- type: Packages
  url: packages/crawlcensus-com-packages.yml
- type: StatusPage
  url: https://crawlcensus.com/site/statuspage.io
- type: SignUp
  url: https://crawlcensus.com/site/register.it
- type: PrivacyPolicy
  url: https://crawlcensus.com/site/privacymanager.io
- type: DomainSecurity
  url: security/crawlcensus-com-domain-security.yml
- type: Website
  url: https://crawlcensus.com/
- type: Documentation
  url: https://crawlcensus.com/guides
- type: APIReference
  url: https://crawlcensus.com/api
- type: Pricing
  url: https://crawlcensus.com/pricing
- type: About
  url: https://crawlcensus.com/about
- type: Support
  url: mailto:hello@crawlcensus.com
- type: GitHubRepository
  url: https://github.com/taylorsmithgg/ai-access-check
- type: GitHubRepository
  url: https://github.com/taylorsmithgg/ai-crawler-registry
apis:
- aid: crawlcensus-com:crawlcensus-com-api
  name: Crawl Census API
  description: Versioned JSON REST API over the Crawl Census store ("Every number on this site, as JSON"). Read endpoints
    for a domain's latest audit (/site/{domain}), live scans (/scan), census aggregates (/census), the AI crawler registry
    (/bots, /bot/{id}), policy change events (/changes), verdicts, facts and compliance. CORS open, 60 requests/day unauthenticated,
    Bearer auth for higher limits, data CC BY 4.0. No machine-readable contract (OpenAPI/AsyncAPI/GraphQL) is published.
  humanURL: https://crawlcensus.com/api
  baseURL: https://crawlcensus.com/api/v1
  tags:
  - AI Crawlers
  - Robots.txt
  - Open Data
  properties:
  - type: Documentation
    url: https://crawlcensus.com/api
x-coverage:
  state: covered
  reason: identity-verified
  detail: Name, description, image and pointers taken from crawlcensus.com pages (title "Crawl Census", /about, /api, /pricing).
    Sitemap /site/<domain> pages are per-domain census reports, not privacy/signup/status pages, so they were not recorded.
    /terms and /privacy return 404.
  evidence:
  - url: https://crawlcensus.com/
    status: 200
  - url: https://crawlcensus.com/about
    status: 200
  - url: https://crawlcensus.com/api
    status: 200
  - url: https://crawlcensus.com/guides
    status: 200
  - url: https://crawlcensus.com/pricing
    status: 200
  - url: https://crawlcensus.com/og/default.png
    status: 200
  - url: https://crawlcensus.com/terms
    status: 404
  - url: https://crawlcensus.com/privacy
    status: 404
  - url: https://github.com/taylorsmithgg/ai-access-check
    status: 200
x-enrichment:
  date: '2026-11-04'
  status: enriched
  artifacts_added: 17
  pass: modular-v1
  engine: identity=anthropic,contract=anthropic,surfaces=anthropic,docs=anthropic,regulatory=anthropic