APIMatic published a head-to-head benchmark of OpenAPI, a docs MCP server and their Context Plugin, and the most interesting sentence in it works against the headline they ran five days earlier. The task was real: add recurring subscription billing to eShopOnWeb, a 13,000-line ASP.NET application, using the Maxio Advanced Billing API, with Claude Opus 4.8 doing the work across four conditions and five runs each. Twenty-four pass/fail checks span correctness, error hygiene, resilience and security. The means separate cleanly — Context Plugin 23.2 of 24, docs MCP 17.2, raw OpenAPI 15.0, an unaided agent 13.8, at a reported Mann–Whitney p of 0.0079. And then: “every reported run completed the functional API task and met the tested API-contract checks, so this benchmark does not show a difference in basic integration success.”
That is a vendor narrowing its own claim in public, and it deserves to be read as such: “the observed advantage is therefore not a happy-path API-success-rate advantage in this sample; it is an operational, maintenance, and workflow difference.” The supporting numbers are the ones that survive scrutiny because they measure behaviour rather than quality — 12 API-learning turns across five runs against 91 for raw OpenAPI, $2.20 of model spend against $5.83, three or fewer hand-maintained wire artifacts against somewhere between 8 and 40, and a median of 799 integrator lines against 963. Five runs per condition is a small sample to attach a p-value to, and the honest read is the direction rather than the magnitude. The single best finding is an anecdote from a control run: an agent located a generated SDK and then replaced its HTTP transport, discarding the retry logic it had just been handed. “A generated client can contain resilient behavior while still being bypassed if the agent lacks guidance about how to configure it.” No contract prevents that. It is an argument for shipping instructions, not just interfaces.
The catalog’s APIMatic record is the toolchain this argument sells. The Validation API is where a specification is told it is inconsistent before an agent inherits the inconsistency, the SDK Generation API produces the client whose retry logic the control run threw away, and the Portals API renders the documentation the 91 learning turns were spent reading.
APIMatic scores 49.6, developing on the Kin Score — down fractionally from 50.3 last week — carried by contract quality at 67.5 and discoverability at 68.5, with governance at 9.8. Agent Readiness holds at 31.9, agent-ready, agent_skills still lit, which remains the thing that makes their argument credible rather than merely marketed. And the two dimensions this benchmark measures most directly are still dark. error_semantics is unlit, on a study that grades error hygiene as one of four check families. idempotency is unlit, on a study that grades resilience. The benchmark is a good-faith piece of work that partly disarms its own press release. The next honest move is to run those twenty-four checks against their own contract.