Agent Harbor Ai Crawlers API
The Ai Crawlers API from Agent Harbor — 1 operation(s) for ai crawlers.
Work with this as data
Every API here is available over the APIs.io API and to AI agents over MCP.
MCP server
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
Tools for apis
7 MCP tools reach this
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.
Call it yourself
curl for this page
This API
curl "https://apis.io/api/v1/apis/agentharbor-agency:agentharbor-agency-ai-crawlers-api"
All apis
curl "https://apis.io/api/v1/apis?limit=25"Discovery needs no key. Ratings and market analysis are Pro.
Get an API key
Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.
A second provider on the same verified email joins the account you already have.
OpenAPI Specification
openapi: 3.2.0
info:
title: Agent Harbor x402 Gateway Ai Crawlers API
version: 1.0.0
x-guidance: 'Pay-per-call business website data for agents. Every paid operation takes one bare domain (e.g. example.com). Call it unpaid to get HTTP 402 with the price; pay with x402 (USDC on Base, PAYMENT-SIGNATURE header) or, where offered, MPP (WWW-Authenticate: Payment, Tempo USDC) and retry. Failed or unusable calls are never charged. Cheapest first: /company, then /scrape; /enrich and /audit are the deeper tiers. Free: /catalog, /llms.txt, /SKILL.md, /.well-known/x402. Public-data routes (SEC EDGAR, recalls, package check, domain verification, PDF) take the query parameters listed in their own entries instead of a domain.'
contact:
name: Agent Harbor LLC
email: anchor@agentharbor.agency
url: https://agentharbor.agency
tags:
- name: Ai Crawlers
paths:
/ai-crawlers:
get:
summary: Ai Crawlers Get
description: 'Given a website domain, reports which AI crawlers may fetch its home page under its robots.txt (RFC 9309): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more, each allowed or blocked, plus Googlebot, listed sitemaps, whether /llms.txt is published and page-level noindex/nosnippet, with fixes. Static check, no browser. Not charged if robots.txt cannot be read (no response, or a bot block).'
operationId: ai_crawlers_get_ai_crawlers_get
responses:
'200':
description: Served and charged.
content:
application/json:
schema: {}
example:
domain: agentharbor.agency
tier: ai-crawlers
price_paid_usdc: 0.01
data:
ai_crawlers:
robots_txt: present
bot_access:
GPTBot: allowed
OAI-SearchBot: allowed
ChatGPT-User: allowed
ClaudeBot: allowed
Claude-SearchBot: allowed
Claude-User: allowed
PerplexityBot: allowed
Google-Extended: allowed
Applebot-Extended: allowed
CCBot: allowed
Bytespider: allowed
Meta-ExternalAgent: allowed
Amazonbot: allowed
googlebot: allowed
sitemaps: []
llms_txt: present
findings:
- id: ai.robots.no_sitemap
severity: low
confidence: confirmed
title: robots.txt does not list a sitemap
fix: 'Add ''Sitemap: https://your-domain/sitemap.xml''.'
evidence: []
- id: ai.llms_txt.present
severity: info
confidence: confirmed
title: llms.txt is present
fix: ''
evidence:
- '# Agent Harbor'
notes: []
provider: Agent Harbor — deterministic audit engine
'402':
description: 'Payment required: x402 v2 PAYMENT-REQUIRED challenge (price, network, payTo); also an MPP `WWW-Authenticate: Payment` challenge (Tempo USDC).'
'422':
description: Invalid domain or a page that cannot be used. Not charged.
'502':
description: Upstream failure. Not charged.
x-payment-info:
protocol: x402
version: 2
scheme: exact
network: eip155:8453
asset: USDC
price_usd: '0.01'
payTo: '0x840C829530DF7B4CCF700Fa3A0060cF2C80b014e'
price:
mode: fixed
currency: USD
amount: '0.01'
protocols:
- x402:
version: 2
scheme: exact
network: eip155:8453
asset: USDC
amount: '10000'
payTo: '0x840C829530DF7B4CCF700Fa3A0060cF2C80b014e'
- mpp:
method: tempo
intent: charge
currency: '0x20C000000000000000000000b9537d11c60E8b50'
chainId: 4217
amount: '10000'
offers:
- method: tempo
intent: charge
currency: '0x20C000000000000000000000b9537d11c60E8b50'
amount: '10000'
parameters:
- name: domain
in: query
required: true
schema:
type: string
description: Bare public hostname, e.g. example.com
example: mailchimp.com
description: Bare public hostname, e.g. example.com
tags:
- Ai Crawlers