CrawlCheck Tool API
The Tool API from CrawlCheck — 4 operation(s) for tool.
Work with this as data
Every API here is available over the APIs.io API and to AI agents over MCP.
MCP server
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
Tools for apis
7 MCP tools reach this
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.
Call it yourself
curl for this page
This API
curl "https://apis.io/api/v1/apis/crawlcheck-io:crawlcheck-io-tool-api"
All apis
curl "https://apis.io/api/v1/apis?limit=25"Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.
Get an API key
Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.
A second provider on the same verified email joins the account you already have.
OpenAPI Specification
openapi: 3.2.0
info:
title: Crawlcheck Io Tool API
version: '1'
contact:
name: CrawlCheck
url: https://crawlcheck.io/policy
email: hello@crawlcheck.io
license:
name: Terms
url: https://crawlcheck.io/policy
x-core-contract:
promise: 'These operations are the core contract. They stay at /api/v1 and keep answering for at least 180 days after any /api/v2 is announced; their stable fields change only in a new major version, and their provisional fields only after 30 days'' notice, per /docs/api/compatibility. Verifying what they return needs none of these calls: crawlcheck-verify.mjs checks a document on your machine against the published keys (/.well-known/http-message-signatures-directory) and the rotation record, read once with --online or saved for --keys. Everything else documented here is useful but secondary, and may change with the notice its fields'' tags promise.'
operations:
- method: GET
path: /api/v1/resolve
operationId: resolveDomainV1
- method: POST
path: /api/v1/resolve
operationId: resolveBatchV1
- method: POST
path: /api/v1/preflight
operationId: preflightV1
- method: GET
path: /api/v1/decisions/{sha256}
operationId: decisionCheckV1
- method: GET
path: /api/v1/revocations
operationId: revocations
- method: GET
path: /api/v1/visitor
operationId: visitorResolveV1
- method: POST
path: /api/v1/visitor
operationId: visitorResolveBatchV1
- method: GET
path: /api/v1/mcp/scan
operationId: mcpScanUrl
- method: POST
path: /api/v1/mcp/lock/check
operationId: mcpLockCheck
list: https://crawlcheck.io/api/v1/core
description: 'Operations tagged Tool across 2 of this provider''s published API definitions: crawlcheck-io-openapi.json, crawlcheck-io-openapi.yml. Each path carries the servers of the definition it was published in.'
servers:
- url: https://crawlcheck.io
tags:
- name: Tool
paths:
/api/tool/wba:
get:
operationId: checkBotKeyDirectory
summary: 'Audit a Web Bot Auth key directory: no redirects, content type, RFC 7638…'
parameters:
- name: agent
in: query
required: true
schema:
type: string
description: The signing agent's host or https origin, e.g. crawlcheck.io
responses:
'200':
description: The directory audit, every step shown
content:
application/json:
examples:
worked:
summary: Captured from production on 2026-09-30 (MDT).
value:
ok: true
mode: directory
agent: https://crawlcheck.io
directory:
url: https://crawlcheck.io/.well-known/http-message-signatures-directory
status: 200
redirected: false
content_type: application/http-message-signatures-directory+json
content_type_ok: true
keys:
- kid: hcIAczGf-8EFBnkunmZlvFWnD2nHHeHeC9cM4BTiBVo
kty: OKP
crv: Ed25519
thumbprint: hcIAczGf-8EFBnkunmZlvFWnD2nHHeHeC9cM4BTiBVo
x: Ny5tAiGdpyVWTm64G1_0g_sTo4ocLL7kqjZ6cK7G5KM
usable: true
bound: true
why: signed this response (sig1)
signatures:
- label: sig1
keyid: hcIAczGf-8EFBnkunmZlvFWnD2nHHeHeC9cM4BTiBVo
tag: http-message-signatures-directory
alg: ed25519
created: 1790813991
expires: 1790814291
components:
- '"@authority";req'
- '"content-digest"'
window: valid for 300 more s
verified: true
why: the signature matches the key and the time window holds
signature_valid: true
content_digest:
present: true
matches: true
algorithm: sha-256
verdict: valid
reasons: []
bound_keys: 1
schema:
type:
- object
- 'null'
properties:
ok:
type:
- boolean
- 'null'
mode:
type:
- string
- 'null'
agent:
type:
- string
- 'null'
directory:
type:
- object
- 'null'
properties:
url:
type:
- string
- 'null'
status:
type:
- number
- 'null'
redirected:
type:
- boolean
- 'null'
content_type:
type:
- string
- 'null'
content_type_ok:
type:
- boolean
- 'null'
keys:
type:
- array
- 'null'
items:
type:
- object
- 'null'
properties:
kid:
type:
- string
- 'null'
kty:
type:
- string
- 'null'
crv:
type:
- string
- 'null'
thumbprint:
type:
- string
- 'null'
x:
type:
- string
- 'null'
usable:
type:
- boolean
- 'null'
bound:
type:
- boolean
- 'null'
why:
type:
- string
- 'null'
additionalProperties: true
signatures:
type:
- array
- 'null'
items:
type:
- object
- 'null'
properties:
label:
type:
- string
- 'null'
keyid:
type:
- string
- 'null'
tag:
type:
- string
- 'null'
alg:
type:
- string
- 'null'
created:
type:
- number
- 'null'
expires:
type:
- number
- 'null'
components:
type:
- array
- 'null'
items:
type:
- string
- 'null'
window:
type:
- string
- 'null'
verified:
type:
- boolean
- 'null'
why:
type:
- string
- 'null'
signature_valid:
type:
- boolean
- 'null'
additionalProperties: true
content_digest:
type:
- object
- 'null'
properties:
present:
type:
- boolean
- 'null'
matches:
type:
- boolean
- 'null'
algorithm:
type:
- string
- 'null'
additionalProperties: true
verdict:
type:
- string
- 'null'
reasons:
type:
- array
- 'null'
items: {}
bound_keys:
type:
- number
- 'null'
additionalProperties: true
additionalProperties: true
x-schema-source: worked-example
'400':
description: No usable agent named
x-codeSamples:
- lang: Shell
label: curl
source: "curl -s \\\n 'https://crawlcheck.io/api/tool/wba?agent=crawlcheck.io'"
security: []
x-auth: public; no credential read
tags:
- Tool
post:
operationId: verifySignedBotRequest
summary: 'Verify a pasted Web Bot Auth request: the agent''s directory, then the request''s…'
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
request:
type: string
description: 'Pasted request headers: Host, Signature-Agent, Signature-Input, Signature, optionally the request line'
agent:
type: string
description: Alternatively, only the agent whose directory to audit
examples:
worked:
summary: The request behind the worked response
value:
agent: crawlcheck.io
responses:
'200':
description: The verdict with every step shown
content:
application/json:
examples:
worked:
summary: Paste request headers in request to check one signed request instead. Captured from production on 2026-09-30 (MDT).
value:
ok: true
mode: directory
agent: https://crawlcheck.io
directory:
url: https://crawlcheck.io/.well-known/http-message-signatures-directory
status: 200
redirected: false
content_type: application/http-message-signatures-directory+json
content_type_ok: true
keys:
- kid: hcIAczGf-8EFBnkunmZlvFWnD2nHHeHeC9cM4BTiBVo
kty: OKP
crv: Ed25519
thumbprint: hcIAczGf-8EFBnkunmZlvFWnD2nHHeHeC9cM4BTiBVo
x: Ny5tAiGdpyVWTm64G1_0g_sTo4ocLL7kqjZ6cK7G5KM
usable: true
bound: true
why: signed this response (sig1)
signatures:
- label: sig1
keyid: hcIAczGf-8EFBnkunmZlvFWnD2nHHeHeC9cM4BTiBVo
tag: http-message-signatures-directory
alg: ed25519
created: 1790813991
expires: 1790814291
components:
- '"@authority";req'
- '"content-digest"'
window: valid for 266 more s
verified: true
why: the signature matches the key and the time window holds
signature_valid: true
content_digest:
present: true
matches: true
algorithm: sha-256
verdict: valid
reasons: []
bound_keys: 1
schema:
type:
- object
- 'null'
properties:
ok:
type:
- boolean
- 'null'
mode:
type:
- string
- 'null'
agent:
type:
- string
- 'null'
directory:
type:
- object
- 'null'
properties:
url:
type:
- string
- 'null'
status:
type:
- number
- 'null'
redirected:
type:
- boolean
- 'null'
content_type:
type:
- string
- 'null'
content_type_ok:
type:
- boolean
- 'null'
keys:
type:
- array
- 'null'
items:
type:
- object
- 'null'
properties:
kid:
type:
- string
- 'null'
kty:
type:
- string
- 'null'
crv:
type:
- string
- 'null'
thumbprint:
type:
- string
- 'null'
x:
type:
- string
- 'null'
usable:
type:
- boolean
- 'null'
bound:
type:
- boolean
- 'null'
why:
type:
- string
- 'null'
additionalProperties: true
signatures:
type:
- array
- 'null'
items:
type:
- object
- 'null'
properties:
label:
type:
- string
- 'null'
keyid:
type:
- string
- 'null'
tag:
type:
- string
- 'null'
alg:
type:
- string
- 'null'
created:
type:
- number
- 'null'
expires:
type:
- number
- 'null'
components:
type:
- array
- 'null'
items:
type:
- string
- 'null'
window:
type:
- string
- 'null'
verified:
type:
- boolean
- 'null'
why:
type:
- string
- 'null'
signature_valid:
type:
- boolean
- 'null'
additionalProperties: true
content_digest:
type:
- object
- 'null'
properties:
present:
type:
- boolean
- 'null'
matches:
type:
- boolean
- 'null'
algorithm:
type:
- string
- 'null'
additionalProperties: true
verdict:
type:
- string
- 'null'
reasons:
type:
- array
- 'null'
items: {}
bound_keys:
type:
- number
- 'null'
additionalProperties: true
additionalProperties: true
x-schema-source: worked-example
'400':
description: Nothing to check
x-codeSamples:
- lang: Shell
label: curl
source: "curl -s -X POST \\\n -H \"content-type: application/json\" \\\n -d '{\"agent\":\"crawlcheck.io\"}' \\\n 'https://crawlcheck.io/api/tool/wba'"
security: []
x-auth: public; no credential read
x-crawlcheck-budget:
request_body_max: 8388608
tags:
- Tool
servers:
- url: https://crawlcheck.io
/api/tool/verify:
post:
operationId: verifyCrawlerLog
summary: Adjudicate crawler identity claims in raw access-log lines against each…
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- lines
properties:
lines:
type: string
description: Up to 200 raw log lines, 20 KB, newline separated
examples:
worked:
summary: The request behind the worked response
value:
lines: '20.171.207.10 - - [30/Sep/2026:22:14:03 +0000] "GET /robots.txt HTTP/1.1" 200 412 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"
203.0.113.9 - - [30/Sep/2026:22:15:41 +0000] "GET / HTTP/1.1" 200 30211 "-" "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"'
responses:
'200':
description: Per-line verdicts and a tally
content:
application/json:
schema:
$ref: '#/components/schemas/VerifyResult'
examples:
worked:
summary: One line from an OpenAI address (verified), one claiming ClaudeBot from a documentation address (spoofed). Captured from production on 2026-09-30 (MDT).
value:
ok: true
counted: 2
tally:
verified: 1
spoofed: 1
unverifiable: 0
no_bot: 0
forged_rate_pct: 50
denominator_note: The rate divides by verified + spoofed only. Requests naming an operator that publishes no feed are excluded - counting them as forged would inflate the number, and counting them as genuine would understate it.
rows:
- line: 20.171.207.10 - - [30/Sep/2026:22:14:03 +0000] "GET /robots.txt HTTP/1.1" 200 412 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.
bot: GPTBot
ip: 20.171.207.10
verdict: verified
why: inside GPTBot's published range
- line: 203.0.113.9 - - [30/Sep/2026:22:15:41 +0000] "GET / HTTP/1.1" 200 30211 "-" "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
bot: ClaudeBot
ip: 203.0.113.9
verdict: spoofed
why: claims ClaudeBot from an address outside every range ClaudeBot's operator publishes
'400':
description: No lines
x-codeSamples:
- lang: Shell
label: curl
source: "curl -s -X POST \\\n -H \"content-type: application/json\" \\\n -d '{\"lines\":\"20.171.207.10 - - [30/Sep/2026:22:14:03 +0000] \\\"GET /robots.txt HTTP/1.1\\\" 200 412 \\\"-\\\" \\\"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot\\\"\\n203.0.113.9 - - [30/Sep/2026:22:15:41 +0000] \\\"GET / HTTP/1.1\\\" 200 30211 \\\"-\\\" \\\"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)\\\"\"}' \\\n 'https://crawlcheck.io/api/tool/verify'"
security: []
x-auth: public; no credential read
x-crawlcheck-budget:
request_body_max: 1048576
tags:
- Tool
servers:
- url: https://crawlcheck.io
/api/tool/robots:
get:
operationId: robotsResolve
summary: Resolve every named answer engine, search index and training crawler against a…
parameters:
- name: domain
in: query
required: true
schema:
type: string
- name: path
in: query
required: false
schema:
type: string
default: /
responses:
'200':
description: 'Per agent: allowed at the path, allowed_root, named, shadowed'
content:
application/json:
schema:
type: object
examples:
worked:
summary: Captured from production on 2026-09-30 (MDT).
value:
ok: true
domain: crawlcheck.io
path: /docs/api
robots:
status: 200
content_type: text/plain
bytes: 1761
readable: true
redirect: ''
measured: true
reason: null
groups: 2
shadowed_groups: 0
tally:
answer:
allowed: 17
total: 17
search:
allowed: 24
total: 24
train:
allowed: 73
total: 73
agents:
- ua: Googlebot-Image
label: Google image crawler
role: search
allowed: true
allowed_root: true
named: false
rules_shadowed: false
- ua: Googlebot-News
label: Google News
role: search
allowed: true
allowed_root: true
named: false
rules_shadowed: false
- ua: Googlebot-Video
label: Google video crawler
role: search
allowed: true
allowed_root: true
named: false
rules_shadowed: false
- ua: Storebot-Google
label: Google Shopping
role: search
allowed: true
allowed_root: true
named: false
rules_shadowed: false
note: 'Only the most specific group applies to a named agent; a Disallow under User-agent: * never applies to an agent that has its own group. Training crawlers are listed, not judged - refusing training is a policy choice.'
'400':
description: No domain
x-codeSamples:
- lang: Shell
label: curl
source: "curl -s \\\n 'https://crawlcheck.io/api/tool/robots?domain=crawlcheck.io&path=/docs/api'"
security: []
x-auth: public; no credential read
tags:
- Tool
servers:
- url: https://crawlcheck.io
/api/tool/llms:
get:
operationId: llmsDraft
summary: Draft an llms.txt from the domain's own homepage and up to 25 declared pages…
parameters:
- name: domain
in: query
required: true
schema:
type: string
- name: format
in: query
required: false
schema:
type: string
enum:
- json
- txt
description: txt returns the file itself
responses:
'200':
description: text plus how many pages were declared, read and carried their own description; text/plain when format=txt
content:
application/json:
examples:
worked:
summary: Captured from production on 2026-09-30 (MDT).
value:
ok: true
domain: treeservicedenverllc.com
site: Tree Service Denver LLC
description: Professional tree service in Denver, CO. Tree removal, trimming, stump grinding & emergency services. Licensed, insured. Call (720) 807-2785.
sitemap_declared: 25
pages_read: 25
pages_with_description: 25
text: '# Tree Service Denver LLC
> Professional tree service in Denver, CO. Tree removal, trimming, stump grinding & emergency services. Licensed, insured. Call (720) 807-2785.
<!-- Draft generated by crawlcheck.io on 2026-10-01 from treeservic…'
at: '2026-10-01T00:19:57.370Z'
note: 'A draft, not a publication. The checks that matter come after: serve it as text/plain, confirm the bare URL returns it, name it in robots.txt.'
schema:
type:
- object
- 'null'
properties:
ok:
type:
- boolean
- 'null'
domain:
type:
- string
- 'null'
site:
type:
- string
- 'null'
description:
type:
- string
- 'null'
sitemap_declared:
type:
- number
- 'null'
pages_read:
type:
- number
- 'null'
pages_with_description:
type:
- number
- 'null'
text: {}
at:
type:
- string
- 'null'
note:
type:
- string
- 'null'
additionalProperties: true
x-schema-source: worked-example
'400':
description: No domain
x-codeSamples:
- lang: Shell
label: curl
source: "curl -s \\\n 'https://crawlcheck.io/api/tool/llms?domain=treeservicedenverllc.com'"
security:
- licenceKey: []
- {}
x-auth: a licence key, or anonymous with the result sent to the given email
tags:
- Tool
servers:
- url: https://crawlcheck.io
components:
schemas:
VerifyResult:
type: object
properties:
tally:
type: object
properties:
verified:
type: integer
spoofed:
type: integer
unverifiable:
type: integer
no_bot:
type: integer
forged_rate_pct:
type:
- number
- 'null'
denominator_note:
type: string
rows:
type: array
items:
type: object
securitySchemes:
licenceKey:
type: apiKey
in: header
name: x-crawlcheck-key
description: 'Your licence key: cc_ followed by 32 hex characters. Sent on every keyed call.'
bearer:
type: http
scheme: bearer
description: The same licence key as a Bearer token. Accepted on /api/v1/* and /api/usage.
externalDocs:
url: https://crawlcheck.io/docs/api
description: Human-readable reference
x-refined-from:
- crawlcheck-io-openapi.json
- crawlcheck-io-openapi.yml