Crawl Census

Crawl Census API

Versioned JSON REST API over the Crawl Census store ("Every number on this site, as JSON"). Read endpoints for a domain's latest audit (/site/{domain}), live scans (/scan), census aggregates (/census), the AI crawler registry (/bots, /bot/{id}), policy change events (/changes), verdicts, facts and compliance. CORS open, 60 requests/day unauthenticated, Bearer auth for higher limits, data CC BY 4.0. No machine-readable contract (OpenAPI/AsyncAPI/GraphQL) is published.

GitHub

Documentation

📖
Documentation
https://crawlcensus.com/api

Other Resources

🔗
APIsJSON
https://artifacts.apis.io/crawlcensus-com/apis.yml

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/crawlcensus-com:crawlcensus-com-api"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

API entry from apis.yml

apis.yml Raw ↑ 
aid: crawlcensus-com:crawlcensus-com-api
name: Crawl Census API
description: Versioned JSON REST API over the Crawl Census store ("Every number on this site, as JSON").
  Read endpoints for a domain's latest audit (/site/{domain}), live scans (/scan), census aggregates (/census),
  the AI crawler registry (/bots, /bot/{id}), policy change events (/changes), verdicts, facts and compliance.
  CORS open, 60 requests/day unauthenticated, Bearer auth for higher limits, data CC BY 4.0. No machine-readable
  contract (OpenAPI/AsyncAPI/GraphQL) is published.
humanURL: https://crawlcensus.com/api
baseURL: https://crawlcensus.com/api/v1
tags:
- AI Crawlers
- Robots.txt
- Open Data
properties:
- type: Documentation
  url: https://crawlcensus.com/api