niche · 1.5 marketcapability industrydomainschema

Robots.txt

Score 19.7 / 100 1 providers +18 via APIs 18 APIs Search apis.io →

Providers using this tag (1)

Ranked by API Evangelist rating — Exemplar and Strong are expanded by default.

Developing 1 Usable, with meaningful gaps to close
AI Crawler IndexAI crawlersweb crawlersUser Agents1 API45.4

APIs with this tag (18)

Ranked by the provider's API Evangelist rating — the Kin Score is scored per provider, not per API, so every API of a provider shares its band. How the rating works →

Exemplar 1 Complete, well-documented, and agent-ready
Hive Civilization Robots.txt APIThe Robots.txt API from Hive Civilization — 1 operation(s) for robots.txt.74.7
Strong 5 Solid coverage with minor gaps
Open Food Facts Robots.txt APIThe Robots.txt API from Open Food Facts — 1 operation(s) for robots.txt.64.5MERCURY Robots Txt APIThe robots-txt API from MERCURY — 1 operation(s) for robots-txt.58.6Blazing Bullet Private Limited Robots.txt APIThe Robots.txt API from Blazing Bullet Private Limited — 1 operation(s) for robots.txt.56.3Pomo Robots.txt APIThe Robots.txt API from Pomo — 1 operation(s) for robots.txt.55.9APIClaw Robots.txt APIThe Robots.txt API from APIClaw — 1 operation(s) for robots.txt.54.6
Developing 8 Usable, with meaningful gaps to close
mcpgate Robots.txt APIThe Robots.txt API from mcpgate — 1 operation(s) for robots.txt.54.21F916 Robots.txt APIThe Robots.txt API from 1F916 — 1 operation(s) for robots.txt.50.9Macaroon Network Robots.txt APIThe Robots.txt API from Macaroon Network — 1 operation(s) for robots.txt.50.5Verified Digital Agents (VDA) Robots.txt APIThe Robots.txt API from Verified Digital Agents (VDA) — 1 operation(s) for robots.txt.46.2University of Amsterdam Robots.txt APIThe Robots.txt API from University of Amsterdam — 1 operation(s) for robots.txt.42.3AgentCheck Robots.txt APIThe Robots.txt API from AgentCheck — 1 operation(s) for robots.txt.41.1HAMi Robots.txt APIThe Robots.txt API from HAMi — 1 operation(s) for robots.txt.40.8AutoMQ Robots.txt APIThe Robots.txt API from AutoMQ — 1 operation(s) for robots.txt.40.3
Thin 2 Limited public surface area
TESSA Marketing & Technology Robots.txt APIThe Robots.txt API from TESSA Marketing & Technology — 1 operation(s) for robots.txt.38.5THORChain Robots.txt APIThe Robots.txt API from THORChain — 1 operation(s) for robots.txt.31.8
Emerging 2 Early or largely undocumented
IRIZ Platform Robots.txt APIThe Robots.txt API from IRIZ Platform — 1 operation(s) for robots.txt.24.0Oauth2-Proxy Robots.txt APIThe Robots.txt API from Oauth2-Proxy — 1 operation(s) for robots.txt.21.6

Companies reaching this through an API (18)

These companies publish an API, specification or operation carrying “Robots.txt” but do not classify their business under it. Listed unranked and kept out of the count above, because one tagged operation is not a statement about what a company does.

1F916 AgentCheck APIClaw AutoMQ Blazing Bullet Private Limited HAMi Hive Civilization IRIZ Platform Macaroon Network mcpgate MERCURY Oauth2-Proxy Open Food Facts Pomo TESSA Marketing & Technology THORChain University of Amsterdam Verified Digital Agents (VDA)

Score breakdown

Frequency
40.8
log-scaled weighted occurrences
Breadth
0.0
spread across providers
Quality lift
47.6
mean composite of providers using it
Cohesion
0.0
strength of nearest seed neighbor

Related tags

llms-txt 8 co-occurrences Tools 6 co-occurrences x402 5 co-occurrences A2A 12 co-occurrences A2A Verified 7 co-occurrences

Where this tag comes from

Api tag18
Provider tag1
Openapi tag18
Openapi op tag18

Work with this as data

Every tag here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for tags

7 MCP tools reach this
  • find_tagsBrowse and filter every tag in the catalog.
  • get_cohortThis tag as a scored cohort — every provider carrying it, with scores.
  • cohort_statsPRO — the distribution across this tag: mean, median, band split, adoption rates.
  • cohort_rankingsPRO — the leaderboard, on composite AND agent-readiness axes.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This tag 
curl "https://apis.io/api/v1/tags/robots-txt"
All tags 
curl "https://apis.io/api/v1/tags?limit=25"
As a scored cohort 
curl "https://apis.io/api/v1/cohorts/tag/robots-txt"
The distribution (Pro) 
curl "https://apis.io/api/v1/cohorts/tag/robots-txt/stats" \
  -H "X-API-Key: $APIS_IO_KEY"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.