# Crawl Census

**Canonical:** https://apis.io/providers/crawlcensus-com/  
**Website:** https://crawlcensus.com/  
**APIs profiled:** 1

Crawl Census (crawlcensus.com) is an independent, unfunded measurement service that audits what AI crawlers and answer engines can actually read on any website. It checks robots.txt policy against 39 tracked AI crawler tokens, sends live requests under real crawler user agents to catch edge blocking, scores JavaScript-free readability, structured data and llms.txt, and publishes a public, CC BY 4.0 record of roughly 30,000 domains and who blocks whom. Access is free via the web, a JSON REST API at /api/v1, and a remote MCP server.

## Kin Score — 29.2 / 100 (thin)

Scored 2026-11-13 under rubric 0.28.0.

| Facet | Score |
|---|---|
| Discoverability | 65.0 |
| Contract Quality | 0.0 |
| Contract Governance | 0.0 |
| Operational Transparency | 47.6 |
| Developer Ergonomics | 40.5 |
| Access Clarity | 44.7 |

Regulatory layer — **Government & Public Sector**: 9.8 (matched via tags).

## Agent readiness — 11.4 (agent-aware)

| Dimension | Value |
|---|---|
| Spec Presence | no |
| Agentic Access | no |
| Reversibility Documented | no |
| MCP Server | documented |
| Auth Clarity | bearer |
| Idempotency | no |
| Error Semantics | documented |
| OpenAPI Examples | no |
| Rate Limit Signal | documented |
| Event Surface Described | no |
| Agent Skills | no |
| Well Known Catalog | no |
| Consent Identity | no |
| Agent Card | no |
| Dry Run Mode | no |
| Delegated Identity | no |
| Protected Resource Metadata | no |
| Dynamic Client Registration | no |
| Agentic Commerce | no |
| Pagination Ratio | na |
| Operation Distinctiveness | na |
| MCP Tool Quality | na |

## Access

Free · Self-serve signup — onboarding: self-serve, pricing: free, trial: no (confidence: high).

## APIs (1)

- **Crawl Census API** — Versioned JSON REST API over the Crawl Census store ("Every number on this site, as JSON"). Read endpoints for a domain's latest audit (/site/{domain}), live scans (/scan), cens...

## MCP servers (1)

- **Crawl Census MCP Server** — Audit how AI crawlers and answer engines can read a website, and query the public census of who blocks whom.

## Security (2)

- **Crawl Census Authentication** — 2 schemes
- **Crawl Census Domain Security** — TLSv1.3 · DMARC

## Plans (1)

- **Crawl Census Plans Pricing**

## Tags

Company, AI Crawlers, Robots.txt, SEO, Web Measurement, Open Data, Llms.txt, MCP

---

Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/providers/crawlcensus-com/). Scores are computed from the provider's own public artifacts under a published rubric.