niche · -3.0 capability resource

Robots.txt

Score 20.0 / 100 2 providers +50 via APIs 51 APIs Search apis.io →

Providers using this tag (2)

Ranked by API Evangelist rating — Exemplar and Strong are expanded by default.

Developing 1 Usable, with meaningful gaps to close
AI Crawler IndexAI CrawlersWeb CrawlersUser Agents1 API46.3
Thin 1 Limited public surface area
Crawl CensusCompanyAI CrawlersSEO1 API29.2

APIs with this tag (51)

Ranked by the provider's API Evangelist rating — the Kin Score is scored per provider, not per API, so every API of a provider shares its band. How the rating works →

Exemplar 1 Complete, well-documented, and agent-ready
Hive Civilization Robots.txt APIThe Robots.txt API from Hive Civilization — 1 operation(s) for robots.txt.78.0
Strong 9 Solid coverage with minor gaps
Open Food Facts Robots.txt APIThe Robots.txt API from Open Food Facts — 1 operation(s) for robots.txt.66.2huggingbay-xyz Robots.txt APIThe Robots.txt API from huggingbay-xyz — 1 operation(s) for robots.txt.63.2MERCURY Robots Txt APIThe robots-txt API from MERCURY — 1 operation(s) for robots-txt.62.3Blazing Bullet Private Limited Robots.txt APIThe Robots.txt API from Blazing Bullet Private Limited — 1 operation(s) for robots.txt.60.1Pomo Robots.txt APIThe Robots.txt API from Pomo — 1 operation(s) for robots.txt.56.9Aveco Robots.txt APIThe Robots.txt API from Aveco — 1 operation(s) for robots.txt.56.8mcpgate Robots.txt APIThe Robots.txt API from mcpgate — 1 operation(s) for robots.txt.56.7CoraPilot Robots.txt APIThe Robots.txt API from CoraPilot — 1 operation(s) for robots.txt.55.3Macaroon Network Robots.txt APIThe Robots.txt API from Macaroon Network — 1 operation(s) for robots.txt.55.1
Developing 19 Usable, with meaningful gaps to close
APIClaw Robots.txt APIThe Robots.txt API from APIClaw — 1 operation(s) for robots.txt.52.61F916 Robots.txt APIThe Robots.txt API from 1F916 — 1 operation(s) for robots.txt.52.1AutoMQ Robots.txt APIThe Robots.txt API from AutoMQ — 1 operation(s) for robots.txt.49.6AI Alt Finder Robots.txt APIThe Robots.txt API from AI Alt Finder — 1 operation(s) for robots.txt.48.0Beste KI Tools Robots.txt APIThe Robots.txt API from Beste KI Tools — 1 operation(s) for robots.txt.47.0aiagentscity-com Robots.txt APIThe Robots.txt API from aiagentscity-com — 1 operation(s) for robots.txt.45.2University of Amsterdam Robots.txt APIThe Robots.txt API from University of Amsterdam — 1 operation(s) for robots.txt.45.2Verified Digital Agents (VDA) Robots.txt APIThe Robots.txt API from Verified Digital Agents (VDA) — 1 operation(s) for robots.txt.45.2AgenticTrade Robots.txt APIThe Robots.txt API from AgenticTrade — 1 operation(s) for robots.txt.44.4Alternativas IA Robots.txt APIThe Robots.txt API from Alternativas IA — 1 operation(s) for robots.txt.44.2Agent Harbor Robots.txt APIThe Robots.txt API from Agent Harbor — 1 operation(s) for robots.txt.43.9HAMi Robots.txt APIThe Robots.txt API from HAMi — 1 operation(s) for robots.txt.43.7Just Domain Robots.txt APIThe Robots.txt API from Just Domain — 1 operation(s) for robots.txt.42.8IA Selecta Robots.txt APIThe Robots.txt API from IA Selecta — 1 operation(s) for robots.txt.42.2AgentCheck Robots.txt APIThe Robots.txt API from AgentCheck — 1 operation(s) for robots.txt.41.6Factanker Robots.txt APIThe Robots.txt API from Factanker — 1 operation(s) for robots.txt.40.9TESSA Marketing & Technology Robots.txt APIThe Robots.txt API from TESSA Marketing & Technology — 1 operation(s) for robots.txt.40.3grokzilla-shop Robots.txt APIThe Robots.txt API from grokzilla-shop — 1 operation(s) for robots.txt.40.0Admit Coach Robots.txt APIThe Robots.txt API from Admit Coach — 1 operation(s) for robots.txt.39.5
Thin 16 Limited public surface area
contactlint-com Robots.txt APIThe Robots.txt API from contactlint-com — 1 operation(s) for robots.txt.38.5journalize-dev Robots.txt APIThe Robots.txt API from journalize-dev — 1 operation(s) for robots.txt.38.2EstateTrace Robots.txt APIThe Robots.txt API from EstateTrace — 1 operation(s) for robots.txt.36.5Boat House Robots.txt APIThe Robots.txt API from Boat House — 1 operation(s) for robots.txt.36.3THORChain Robots.txt APIThe Robots.txt API from THORChain — 1 operation(s) for robots.txt.35.1lastseen-dev Robots.txt APIThe Robots.txt API from lastseen-dev — 1 operation(s) for robots.txt.33.8sarnai-dev Robots.txt APIThe Robots.txt API from sarnai-dev — 1 operation(s) for robots.txt.33.7krimskrams-xyz Robots.txt APIThe Robots.txt API from krimskrams-xyz — 1 operation(s) for robots.txt.32.8Glyph Robots.txt APIThe Robots.txt API from Glyph — 1 operation(s) for robots.txt.30.9AgentPay Robots.txt APIThe Robots.txt API from AgentPay — 1 operation(s) for robots.txt.29.8BuySignal Robots.txt APIThe Robots.txt API from BuySignal — 1 operation(s) for robots.txt.29.3Crawl Census APIVersioned JSON REST API over the Crawl Census store ("Every number on this site, as JSON"). Read endpoints ...29.2HinduNumerology.com Robots.txt APIThe Robots.txt API from HinduNumerology.com — 1 operation(s) for robots.txt.29.1INITE SOLUTIONS Robots.txt APIThe Robots.txt API from INITE SOLUTIONS — 1 operation(s) for robots.txt.28.0Dados B3 Robots.txt APIThe Robots.txt API from Dados B3 — 1 operation(s) for robots.txt.27.9IRIZ Platform Robots.txt APIThe Robots.txt API from IRIZ Platform — 1 operation(s) for robots.txt.26.3
Emerging 6 Early or largely undocumented
Cathedral AI Robots.txt APIThe Robots.txt API from Cathedral AI — 1 operation(s) for robots.txt.24.6Dear Human Robots.txt APIThe Robots.txt API from Dear Human — 1 operation(s) for robots.txt.23.4JNexum Robots.txt APIThe Robots.txt API from JNexum — 1 operation(s) for robots.txt.23.2Oauth2-Proxy Robots.txt APIThe Robots.txt API from Oauth2-Proxy — 1 operation(s) for robots.txt.23.1艾赋 AIFU Robots.txt APIThe Robots.txt API from 艾赋 AIFU — 1 operation(s) for robots.txt.20.7GenMB Robots.txt APIThe Robots.txt API from GenMB — 1 operation(s) for robots.txt.16.7

Companies reaching this through an API (50)

These companies publish an API, specification or operation carrying “Robots.txt” but do not classify their business under it. Listed unranked and kept out of the count above, because one tagged operation is not a statement about what a company does.

1F916 Admit Coach Agent Harbor AgentCheck AgenticTrade AgentPay AI Alt Finder aiagentscity-com Alternativas IA APIClaw AutoMQ Aveco Beste KI Tools Blazing Bullet Private Limited Boat House BuySignal Cathedral AI contactlint-com CoraPilot Dados B3 Dear Human EstateTrace Factanker GenMB Glyph grokzilla-shop HAMi HinduNumerology.com Hive Civilization huggingbay-xyz IA Selecta INITE SOLUTIONS IRIZ Platform JNexum journalize-dev Just Domain krimskrams-xyz lastseen-dev Macaroon Network mcpgate MERCURY Oauth2-Proxy Open Food Facts Pomo sarnai-dev TESSA Marketing & Technology THORChain University of Amsterdam Verified Digital Agents (VDA) 艾赋 AIFU

Score breakdown

Frequency
49.8
log-scaled weighted occurrences
Breadth
0.0
spread across providers
Quality lift
37.8
mean composite of providers using it
Cohesion
0.0
strength of nearest seed neighbor

Related tags

Llms.txt 36 co-occurrences AGENTS.md 7 co-occurrences

Where this tag comes from

Api tag51
Provider tag2
Openapi tag50
Openapi op tag53

Work with this as data

Every tag here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for tags

7 MCP tools reach this
  • find_tagsBrowse and filter every tag in the catalog.
  • get_cohortThis tag as a scored cohort — every provider carrying it, with scores.
  • cohort_statsPRO — the distribution across this tag: mean, median, band split, adoption rates.
  • cohort_rankingsPRO — the leaderboard, on composite AND agent-readiness axes.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This tag 
curl "https://apis.io/api/v1/tags/robots-txt"
All tags 
curl "https://apis.io/api/v1/tags?limit=25"
As a scored cohort (paid plan)
The distribution (Pro)

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.