MERCURY

MERCURY Robots Txt API

The robots-txt API from MERCURY — 1 operation(s) for robots-txt.

Documentation GitHub OpenAPI

Operations 1

GET /buy/robots AI Crawler Permission Audit — signed robots.txt / llms.txt / ai.txt verdict… #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/mercury-hq-com-robots-txt-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: MERCURY x402 storefront Robots Txt API
  version: '1'
  x-spec: mercury-storefront/1
  description: Agent-payable resources over HTTP 402 (x402). LIVE — Base mainnet, real USDC, no token. Only currently-deliverable routes are listed (web-fetch is the live paid SKU; mints are gated off). Free discovery at /.well-known/x402, /x402/discovery, /catalog, and /manifest.
servers:
- url: https://network.mercury-hq.com
tags:
- name: robots-txt
paths:
  /buy/robots:
    get:
      summary: AI Crawler Permission Audit — signed robots.txt / llms.txt / ai.txt verdict…
      description: 'Domain → signed, timestamped per-AI-crawler allow/block audit from robots.txt + llms.txt + ai.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, …). Deterministic, no LLM. EU-AI-Act / TDM opt-out evidence: the signed verdict proves the crawler policy as it stood at fetch time.'
      operationId: buy_cited_robots
      tags:
      - robots-txt
      responses:
        '200':
          description: Delivered after the x402 payment settles on Base mainnet.
          content:
            application/json:
              schema:
                type: object
                properties:
                  ok:
                    type: boolean
                    description: true on success; false on an honest failure (still delivered, never charged for a stub)
                  url:
                    type: string
                    description: origin audited (https://host)
                  status:
                    type: integer
                    description: HTTP status of the robots.txt fetch
                  text:
                    type: string
                    description: canonical JSON the signed attestation is computed over (data, keys sorted)
                  fetchedAt:
                    type: string
                    description: ISO-8601 fetch time — the 'as-of' moment the signed policy snapshot proves
                  data:
                    type: object
                    description: the structured audit
                    properties:
                      version:
                        type: string
                      host:
                        type: string
                      origin:
                        type: string
                      evaluatedPath:
                        type: string
                      policies:
                        type: object
                        description: raw state of each crawler-policy file at fetch time
                        additionalProperties: false
                        properties:
                          robots.txt:
                            type: object
                            additionalProperties: true
                          llms.txt:
                            type: object
                            additionalProperties: true
                          ai.txt:
                            type: object
                            additionalProperties: true
                      verdicts:
                        type: array
                        description: per-AI-agent allow/block verdict for evaluatedPath, with the decisive rule cited
                        items:
                          type: object
                          properties:
                            agent:
                              type: string
                            vendor:
                              type: string
                            purpose:
                              type: string
                            verdict:
                              type: string
                              enum:
                              - allowed
                              - blocked
                            reason:
                              type: string
                            decidedBy:
                              type: string
                            rule:
                              type:
                              - string
                              - 'null'
                      summary:
                        type: object
                        properties:
                          allowed:
                            type: integer
                          blocked:
                            type: integer
                          blockedAgents:
                            type: array
                            items:
                              type: string
                          sitemaps:
                            type: array
                            items:
                              type: string
                          hasLlmsTxt:
                            type: boolean
                          hasAiTxt:
                            type: boolean
                          hasRobotsTxt:
                            type: boolean
                  error:
                    type: string
                    description: present only when ok:false
                required:
                - ok
                - url
                additionalProperties: true
        '402':
          description: Payment Required — retry with an x402-signed payment (e.g. x402-fetch). Terms are in the challenge body (x402 v1).
      parameters:
      - name: domain
        in: query
        required: true
        schema:
          type: string
          maxLength: 2048
          description: domain or URL to audit (e.g. example.com or https://example.com/page); only the origin is used
        description: domain or URL to audit (e.g. example.com or https://example.com/page); only the origin is used
      - name: path
        in: query
        required: false
        schema:
          type: string
          maxLength: 1024
          description: optional path to evaluate the verdict for (default '/', the whole-site question)
        description: optional path to evaluate the verdict for (default '/', the whole-site question)
      x-payment-info:
        protocols:
        - x402
        scheme: exact
        price: $0.005
        currency: USDC
        network: base
        networkCaip2: eip155:8453
        testnet: false
        asset: '0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913'
        payTo: '0xe10B9d44e72A29B9c19da02981FFCd875308e3C1'
        facilitator: https://api.cdp.coinbase.com/platform/v2/x402
        x402Version: 1
      x-x402:
        scheme: exact
        price: $0.005
        currency: USDC
        network: base
        networkCaip2: eip155:8453
        asset: '0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913'
        payTo: '0xe10B9d44e72A29B9c19da02981FFCd875308e3C1'
        facilitator: https://api.cdp.coinbase.com/platform/v2/x402
        testnet: false
        maxTimeoutSeconds: 60
x-payment-info:
  protocols:
  - x402
  network: base
  currency: USDC
  payTo: '0xe10B9d44e72A29B9c19da02981FFCd875308e3C1'
  facilitator: https://api.cdp.coinbase.com/platform/v2/x402
  testnet: false
  spec: mercury-storefront/1