Agent-Readiness Playbook — what we did to apis.io

A portable checklist of the agent-friendly surface we built into the apis.io network. Each item lists what, why, the standard it implements, and where to look in this repo if you want to copy the implementation onto another site.

The goal: a site that both humans and autonomous agents can navigate without scraping HTML or guessing structure.

1. /.well-known/api-catalog — the canonical machine entrypoint

What: The network publishes /.well-known/api-catalog (and a .json mirror) at the apex as an RFC 9264 linkset. Each entry has an anchor (the page URL), a title, and zero or more of service-desc (machine-readable specs — OpenAPI / AsyncAPI / Postman), service-doc (human docs), and describedby (schemas).

Why: Agents discovering a new domain check /.well-known/ first. A linkset is the smallest possible “table of contents” they can consume without an HTML parser.

Spec: RFC 9727 — /.well-known/api-catalog, RFC 9264 — Linkset format.

Where:

Port checklist: generate a linkset JSON for every public collection on your site → write it to /.well-known/api-catalog → set Content-Type: application/linkset+json (see §2).

2. Edge functions — agent-readiness the static origin can’t provide

What: Two CloudFront Functions sit in front of apis.io and do the jobs a static origin cannot:

  1. Markdown content negotiation (viewer-request). When Accept prefers text/markdown, the same clean URL serves the page’s markdown twin instead of the HTML. /providers/stripe/ answers as a 10 KB machine-readable document to a client that asks for one and as the full page to a browser.
  2. RFC 8288 Link headers (viewer-response) on every HTML response — rel="api-catalog", rel="ard", rel="ai-catalog", rel="describedby" for the onboarding descriptor and the MCP server card, rel="sitemap", and rel="alternate" for llms.txt. An agent finds the machine surface without parsing HTML.
  3. Vary: Accept on both representations, so a cache between us and the reader never hands one to a client that asked for the other.
  4. Correct content types on the discovery documents — application/linkset+json on the api-catalog, which a plain static host serves as text/plain.

Why: Static hosts can’t content-negotiate or set content types per path. Two small edge functions graft both onto any static site, and negotiation is the piece that matters most: it is the difference between a coding agent pulling a 10 KB document and pulling a 288 KB page to find the same facts.

How it stays cheap: the negotiation happens on viewer-request, before the cache lookup, by rewriting the request to a different object. The two representations get different cache keys for free — no Accept in the cache policy, so no fragmentation from a header every browser spells differently.

Spec: RFC 8288 Link headers, RFC 9110 §12 content negotiation, RFC 9421 HTTP Message Signatures.

Where: apis-io-aws/cloudfront/url-rewrite.js, cloudfront/link-headers.js.

Port checklist: both functions are one file each and under 10 KB, which is the CloudFront Function size limit. Swap the hostnames and the twin-filename map. On Cloudflare the same two jobs fit in one Worker.

3. robots.txt with explicit AI signals

What: Beyond the standard User-agent: * / Allow: /, the file carries two explicit machine-readable permission signals:

Content-Signal: search=yes, ai-input=yes, ai-train=yes
Content-Usage: search=y, ai-input=y, ai-train=y

Plus a # NOTICE: comment for any agent that parses the file as plain text.

Why: “Crawl this” and “use this for AI input/training” are now distinct grants. The Cloudflare and AIPREF signals make consent explicit at the protocol layer rather than burying it in terms of service.

Spec: Cloudflare Content Signals, draft-ietf-aipref-attach (Content-Usage).

Where: apis.io/robots.txt.

Port checklist: decide what you grant (search / ai-input / ai-train) and copy the two header lines into your robots.txt. Make the comment block explicit.

4. Agent Skills — discoverable how-to packs

What: /skills/ publishes a skill index plus three skill files in the Agent Skills format:

Each is a single SKILL.md with a description and operational instructions. An index lives at skills/index.json with a $schema reference.

Why: Skills are the closest thing the agent ecosystem has to a published “operating manual.” Shipping them means agents don’t have to infer how to use your site — they can read instructions you wrote.

Spec: agentskills.io schema.

Where: apis.io/skills/.

Port checklist: write one skill per common workflow on your site. Index them in a skills/index.json with the agentskills.io $schema. Link to it from your homepage and from Link headers if relevant.

5. Markdown alongside the HTML

What: Provider pages, API pages and Agent Skills each ship a text/markdown twin beside the HTML. Agents reach it two ways: send Accept: text/markdown to the ordinary page URL, or follow the <link rel="alternate" type="text/markdown"> the HTML advertises to the twin’s own .md URL.

Both doors matter, because different clients use different ones. Measured over 33 days on our own traffic: every crawler is 100% direct — ClaudeBot, Amazonbot, GPTBot, Meta and OAI-SearchBot all follow the <link rel="alternate"> to the .md URL, and not one of them negotiates. Coding agents are the opposite: Claude Code negotiates and never constructs the .md URL. Ship only one door and you are invisible to half the population.

Why: agents waste tokens parsing HTML. A provider twin is ~10 KB against a ~288 KB page — the same facts at a fraction of the context.

Scope it honestly. Twins exist where a page is a record — /providers/<slug>/, /apis/<provider>/<api>/, /skills/<name>/ — not on index and landing pages, which are navigation and have nothing to say in markdown. Negotiation falls through to HTML anywhere a twin does not exist, so a client that prefers markdown on a page without one still gets the page.

Where: twins are generated by the site build; negotiation is §2’s viewer-request function.

Port checklist: publish index.md next to index.html for record pages, advertise it with <link rel="alternate" type="text/markdown">, and add negotiation at the edge. The <link> alone gets you the crawlers; negotiation gets you the coding agents.

6. APIs.json — the site’s own machine-readable identity

What: The site identifies itself at /apis.json and /apis.yml using the APIs.json specification — name, description, image, tags, network siblings, maintainers, and common properties (Website, Portal, Blog, Support, ToS, Privacy, GitHub, etc).

Why: It is the machine-readable “who are you, and what do you offer” file. Agents looking for the operational metadata of a site (support contact, blog feed, privacy policy URL) read this one file instead of crawling.

Spec: APIs.json 0.20.

Where: apis.io/apis.json, apis.io/apis.yml.

Port checklist: publish an apis.json at your root. Fill in name, description, tags, common (with at least Website, Support, ToS, Privacy), and maintainers. Use the latest spec version.

7. One namespace per resource type

What: Content is split by resource type rather than piled into one undifferentiated site — /providers/, /apis/, /schemas/, /asyncapis/, /jsonld/, /rules/, /vocabularies/, /tags/, /plans/, /rate-limits/, /finops/ and more. Each namespace has one job, one schema, and its own sitemap.

These began as separate subdomains and were consolidated onto the apex, with the legacy hostnames 301-redirecting to the matching path. If you are choosing today, choose paths: the split is what buys you the crawler and agent benefits, and the extra hostnames bought nothing that a path prefix does not, while costing a certificate entry, a DNS record and a redirect rule each.

Why: Splitting by type is friendly to crawlers (smaller, focused sitemaps), friendly to agents (one namespace, one shape), and friendly to humans (the navigation hierarchy is the URL).

Port checklist: if your site has distinct resource types, give each a clearly-prefixed path (/providers/, /apis/, /schemas/) and its own sitemap. This is the single highest-leverage structural decision in the playbook.

8. JSON-LD contexts published per resource

What: Every provider publishes one or more JSON-LD @context documents at json-ld.apis.io/jsonld/{provider}/{context}/. These give the provider’s vocabulary a stable, dereferenceable IRI.

Why: JSON-LD is how agents map your terms to standard vocabularies (Schema.org, etc). Publishing the context as a real URL lets downstream consumers cite or fetch it.

Where: json-ld/_jsonld/.

Port checklist: for any structured data your site publishes, ship a JSON-LD context at a stable URL. Reference standard vocabularies (Schema.org, DCAT, etc) where they apply.

9. Source widget — show the raw machine data on the human page

What: Every detail page renders the upstream YAML/JSON next to the rendered content, with Prism syntax highlighting, line numbers, search, format toggle (YAML ↔ JSON), and a download button. Humans see exactly what agents see.

Why: Trust. Humans evaluating an agent-readable site can verify the machine data without curl. Also reinforces the “machine-first” framing — the structured data isn’t hidden behind the prose, it is the page.

Where: network/_includes/source-widget.html.

Port checklist: on any page that has underlying structured data, show it. Hide it behind a collapsed disclosure if you must, but ship it.

10. Sitemap.xml per subdomain — the boring fallback

What: Each subdomain has its own sitemap.xml listing every page. Advertised in Link: rel="sitemap" headers and in robots.txt.

Why: Some agents and most search crawlers still rely on sitemaps. The api-catalog is the modern path; sitemap is the universal fallback.

Where: apis.io/sitemap.xml.

Port checklist: generate one per subdomain (Jekyll’s jekyll-sitemap plugin handles this) and reference it from robots.txt and Link headers.

11. Stable, slugged URLs for every resource

What: Every resource lives at a deterministic permalink — /{collection}/{slug}/. Slugs are derived from a canonical identifier (aid), with a deterministic fallback when missing. No bare or empty path segments.

Why: Agents cache and cite URLs. URL instability breaks every downstream system (citations, embeddings, training corpora, link graphs). This is non-negotiable.

Where: network/scripts/build.py — see extract_api_slug and the slug fallback at the call site.

Port checklist: every public resource gets a stable URL. Slugs derive from a deterministic identifier. Have a fallback that produces a usable slug even when the source data is incomplete — and test that the fallback doesn’t produce empty filenames.

12. Apex /.well-known/api-catalog — the network root

What: apis.io/.well-known/api-catalog is the network’s one RFC 9264 linkset, anchoring every namespace and the documents that describe the site itself. An agent that lands on the apex gets a one-hop map of the whole network without crawling it.

Why: agent-readiness scanners check the apex first. Discovery documents scattered across a site are worth much less than one document at the address everything checks — put the map where the agent lands.

Where: apis.io/.well-known/api-catalog, apis.io/.well-known/api-catalog.json.

Port checklist: publish the catalog at the apex, whatever your internal structure. One linkset entry per namespace, with service-desc pointing at the machine-readable description of each.

13. /.well-known/agent-skills/index.json — Cloudflare’s discovery RFC

What: Mirror the /skills/index.json file at /.well-known/agent-skills/index.json, upgraded to the v0.2.0 schema with type and sha256 for each skill entry.

Why: Agent Skills Discovery RFC v0.2.0 expects the well-known path. The Link: rel="agent-skills" header is the modern advertisement; the well-known path is the fallback agents check directly.

Where: apis.io/.well-known/agent-skills/index.json.

Port checklist: add type and sha256 to every entry, publish at the well-known path, and keep the legacy /skills/index.json in sync if you advertised it via Link header.

14. /.well-known/mcp/server-card.json — MCP Server Card (SEP-2127)

What: A stub Model Context Protocol Server Card declaring the planned MCP surface (search, fetch-spec, describe-provider) and pointing at the discovery endpoints (api-catalog, agent-skills, apis.json) as the interim consumption path.

Why: Agents looking for MCP servers check /.well-known/mcp/server-card.json. Publishing a stub now lets them discover the network, learn the planned tool surface, and consume the same data through the api-catalog until a hosted MCP transport ships.

Where: apis.io/.well-known/mcp/server-card.json.

Port checklist: if you don’t yet host an MCP transport, publish a card with transport.status: "planned" plus a discovery block that points at your existing machine surface. Never fabricate a transport URL — agents will fail closed.

15. /.well-known/http-message-signatures-directory — Web Bot Auth directory

What: A JWKS at /.well-known/http-message-signatures-directory (per draft-meunier-web-bot-auth-architecture). Currently an empty keys array because apis.io doesn’t sign outbound HTTP, plus a note describing intent and pointing at the inbound observability layer in the Worker.

Why: Standing up an empty-but-valid directory now means we can add keys later without changing any agent expectation about the URL. The URL is the contract; the key set is allowed to be empty until you actually sign outbound traffic.

Where: apis.io/.well-known/http-message-signatures-directory.

Port checklist: publish a valid JWKS even if keys is []. Document in the file (or alongside it) whether you actually sign outbound bot traffic.

16. WebMCP — in-browser tool exposure

What: A shared runtime registers tools with the browser’s modelContext, so an MCP-capable browser can invoke a page’s actions without reading the DOM. Two surfaces carry tools today:

Why: WebMCP is the in-browser counterpart to the server-side MCP server card. It gives agents a structured handle on the interactive surface so they don’t have to scrape results out of HTML — and, on /stack/, on state that exists nowhere else.

Where: network/_includes/webmcp.html (the runtime, included from _shared/_layouts/default.html inside <head>), with tool declarations in network/index.html and network/stack.html.

Two design rules this surface earns, both worth copying:

  1. Register in ONE place, and feature-detect in spec order. WebMCP is a moving draft: the W3C Web ML CG draft of 2026-05-27 put the getter on document.modelContext with registerTool(tool), while Chrome ≤149 exposed navigator.modelContext.provideContext({tools}) and Chrome deprecated that surface in 150 (origin trial 149–156). A feature-detected call against a draft that has moved degrades to a silent no-op — no error, no console warning, nothing in a diff. One shared runtime means following the spec is a one-file change; detect document first and treat navigator as an explicit legacy fallback.

  2. Instrument from day one, or shipping early buys you nothing. Being first is only worth something if you can say what happened. Every registration and tool call beacons to POST /webmcp/beacon, and capability is sampled at 1% on the negative side so there is a real denominator without a write on every pageview. Read it with the Logs Insights queries in apis-io-aws/WEBMCP-QUERIES.md. The sampling rule matters more than it looks: every adoption figure is sum(weight), never count().

Also worth stealing: the spec has execute() resolve to an MCP content array, not a plain object. The runtime normalizes the return so tool authors keep writing plain objects and cannot get the wire shape wrong.

Port checklist: wrap your site’s primary action (search, query, submit) in a tool, and prefer the actions that have no server-side API behind them — that is where WebMCP adds a capability rather than duplicating one. Reuse the same handlers your DOM event listeners call so the tool and the UI cannot diverge. Register through one shared runtime, never inline per page. Mark state-mutating tools as such in their descriptions. Instrument from day one.

Honest status (2026-09): no mainstream agent calls WebMCP tools yet — Claude, ChatGPT Agent, Gemini and Perplexity still read pages through the DOM or screenshots. We ship it to be early and to measure the arrival, and the beacon is what will tell us whether it ever does arrive.

17. RFC-anchored, not vendor-anchored

What: Wherever there was a choice, we picked the IETF/W3C standard over the vendor convention:

Why: Standards outlive vendors. Building on RFCs means the surface remains agent-readable when the agents change.

Port checklist: if a vendor-specific format and an RFC exist for the same job, default to the RFC. Add the vendor format alongside only if it materially expands reach.

18. Action-for-access gate — a pattern, not a live surface on apis.io

Status first, because it is the part worth being exact about: apis.io’s machine representations are currently OPEN. /providers/<slug>/index.md, apis.json and the discovery documents all answer 200 with no key. The gate below is a design we built and ran at the edge, kept here because the shape is portable and the trade-off is the interesting part — not because apis.io is enforcing it today.

The pattern: an agent performs one cheap, human-meaningful action — star a repo — and POSTs {challenge_id, github_username, operator_email} to an unlock endpoint to receive an opaque key. The key is then required on the expensive surface (machine representations) while HTML, /.well-known/*, /skills/*, apis.json, sitemap.xml and robots.txt stay open.

The refusal uses RFC 9457 Problem Details and advertises the unlock flow in a Link header, so a well-behaved agent can follow it without a human:

HTTP/1.1 402 Payment Required
Content-Type: application/problem+json
Link: <https://apis.io/unlock/challenge>; rel="http://apis.io/rels/unlock"

Why the shape is right even if you never turn it on: agents drive most of the egress cost on a free static site, and a single human-friendly action gives the maintainer a public traction signal in exchange for the expensive surface. Keeping discovery open is not optional — gate /.well-known/*, the sitemap or apis.json and you fall off every agent-readiness scanner, which costs you more than the bandwidth ever did.

Spec: RFC 9457 Problem Details, RFC 8288 Link headers. The 402 status is reused per HTTP semantics (RFC 9110 §15.5.2).

Port checklist: decide what you gate (machine reps, never discovery), pick the action the unlock requires (star, follow, payment, signed gist), return 402 + Problem Details + a Link: rel="…/unlock", and log every gate event to a queryable store. Publish plainly whether the gate is on — an access policy readers cannot verify is worse than no policy.

Quick-start order of operations

If you are starting from a static site and want to apply this playbook, the highest-leverage steps in order:

  1. Ship /apis.json describing your site (1 hour).
  2. Ship /.well-known/api-catalog listing every public resource (1 day — generator + RFC 9264 format).
  3. Update robots.txt with explicit Content-Signal / Content-Usage (10 minutes).
  4. Add edge functions for Link headers + content-type fixes + markdown negotiation (2–4 hours). One Cloudflare Worker, or two CloudFront Functions — the jobs are the same either way.
  5. Write 2–3 Agent Skills covering your most common workflows (a few hours each).
  6. Audit URL stability — fix any empty slugs, bare paths, or non-deterministic permalinks (variable).
  7. Add JSON-LD contexts for your structured data (variable).
  8. Add a source widget to detail pages (1 day).

Steps 1–4 get you 80% of the way to agent-readable. Steps 5–8 are what separates “agents can read this” from “agents want to read this.”

What we explicitly chose not to do

References