Coasty says computer-use benchmarks lie, then cites its own

Coasty says computer-use benchmarks lie, then cites its own

Coasty’s blog says computer use API integration is broken and most people are too scared to say it, and it leads with its hardest number: computer-use agents “hit 85% on benchmarks but fail 80% of real workflows,” which Coasty attributes to June 2026 research. Its list of why is specific. Screen-based integrations need redeploying when layouts change, browser automation breaks on lazy loading, error handling is “often non-existent” so one failure cascades, and token costs explode when agents retry and fail. The line that lands: “Your API integration is probably delivering 20% of the value everyone promises.”

The tension is inside the post itself. Coasty spends its opening paragraphs arguing that OSWorld scores are synthetic and “designed to be easy enough to pass, not accurate enough to matter,” and then, under a heading that reads “Why Coasty Actually Works (And Competitors Don’t),” cites its own 85.6% on OSWorld and 82.81% independently verified on the OSWorld-Verified leaderboard. Those are vendor figures on the benchmark the vendor just called meaningless. What survives is the engineering argument: retries, error handling and guardrails decide whether a computer-use agent works in production, more than the model’s score does.

That argument is one the catalog can actually check, because Coasty publishes an API for it. Coasty on apis.io carries eight API pages from a published OpenAPI 3.1 contract of 72 operations. The cascade-and-retry problem the post describes lives in the Coasty Runs API and Coasty Workflows API; the desktop the agent drives is the Coasty Machines API and Coasty Sessions API; and the screen-reading step is the Coasty Predict API. The agentic access profile counts 72 operations, 42 acting and five flagged human-in-the-loop, which is the guardrail the post asks for, written into the contract.

The measurement backs the retry argument more than the benchmark. Coasty scores 59.4 on the Kin Score, strong, with access clarity at 76.3 and contract quality at 61.6. Agent Readiness is 44.0, and the lit dimensions are the ones the post’s failure list needs: idempotency, error semantics, reversibility documented and rate limit signal. A retry that cannot double-run is what an Idempotency-Key contract buys, and Coasty has one. The weak spot is contract governance at 4.5. And the catalog record describes a first-party MCP server of 26 tools while the MCP server dimension reads unlit, which says the server is not yet discoverable the way the scorer looks for it.

← Our CI/CD page ranks Salesforce third and leaves out CircleCI, Jenkins and Buildkite
Kong says inventory every API first, and its own catalog record shows 139 of them →