Braintrust traces MCP servers, and scores 23.7 on operational transparency

Braintrust traces MCP servers, and scores 23.7 on operational transparency

Braintrust’s post Understand how agents use your MCP server starts from an uncomfortable fact for anyone shipping MCP: “You don’t always control the agent using it, though, or have access to the conversations.” A tool call can succeed and still hand the agent the wrong data. Braintrust’s example is a query tool that ignores an environment filter, so a request for production traffic comes back with staging rows mixed in and no error anywhere. The new feature traces each MCP call, recording inputs, results, timing and errors, through the JavaScript and Python SDKs, and it instruments both the official MCP SDKs and third-party implementations such as FastMCP.

There are no marketing figures in this one, which makes the claim easy to test on its merits. The two optional additions are the interesting part. A telemetry.intent field on a tool’s schema lets the agent say why it is calling, and a send_feedback tool lets the agent report a confusing response along with the tool name and the arguments that produced it. Both turn the agent into a witness. “Patterns” then clusters recurring problems across traces. The advice that survives independent of the product is simple and correct: a 200 is not evidence that the agent got what it asked for, so record the request next to the answer.

The catalog record for Braintrust is 29 API pages, and the story lands on a few of them. Traced calls are read in Logs, which is the Braintrust Logs API; the evals the post suggests building from representative calls run through the Braintrust Evals API and the Braintrust Datasets API; and the servers themselves are registered through the Braintrust MCP Servers API. The agentic access profile counts 208 operations, 89 of them acting.

Braintrust’s Kin Score is 48.0, developing, and the facet that drags it is operational transparency at 23.7. That is the measurement of whether a provider publishes its own status, changelog and limits, and it is the lowest facet on a company that sells visibility into other people’s systems. Contract governance sits at 13.6. Agent Readiness is the better number, 52.4, with MCP server, delegated identity, protected resource metadata, dynamic client registration and OpenAPI examples all lit. Error semantics is unlit, which is the exact gap this post is about: the agent cannot tell a wrong answer from a right one when the contract does not describe what wrong looks like.

← AsyncAPI on apis.io is 3,664 documents, and one crypto exchange wrote 41 of them
Data Enrichment Is Built for Agents and 1.6 Percent of It Documents Consent →