LiteLLM has published a benchmark of decision models used as rerankers. In JevRank: decision models through LiteLLM’s rerank API, the team ran 6,000 document decisions through its rerank endpoint with TypeSafe Jev 1.13.0 on the SciFact set, 300 queries with 20 BM25 candidates each. nDCG@10 rose from 0.686 to 0.768 at an estimated $0.575 per 1,000 queries, with a 0.50 s median and 1.15 s p95, and Jev put a relevant document first for 68% of queries against 55% for BM25. The headline figure is a different one: in reference results run through OpenRouter, Perplexity Decider Flash and Cohere Rerank 4 Pro both scored 0.775, at $73 versus $2,500 per million queries, about 34 times cheaper.
This is the rare benchmark post that marks its own limits, and the caveats are the best part of it. The 34x comparison is reference data, “not our LiteLLM run.” Costs exclude gateway overhead. Jev 1.14’s figure is projected from a rounded per-1K price. The reference run’s candidate sets “were not verified to match this run.” And the line every model leaderboard should carry: “Small score gaps do not establish a quality win.” What survives is LiteLLM’s own measurement — a 0.082 nDCG gain over BM25 for well under a dollar per thousand queries — and a recorded regression shown alongside the improvements.
The story touches one page in the catalog for LiteLLM, the LiteLLM Rerank API, which carries the /rerank path the benchmark called. It has a twin: the LiteLLM Rerank API appears a second time under a prefixed slug, and the same doubling runs through Completions, Embeddings, Batches, Audio, Moderations, OCR and Vector Stores, so the 33 API pages describe noticeably fewer distinct surfaces. Missing entirely is the endpoint the page’s own banner announces — one /v1/decisions route for Jev, Clef and other decision models. Nothing on record mentions it.
The Kin Score is 34.1, thin, with developer ergonomics at 50.0, discoverability at 51.8 and contract governance at 13.6. Agent Readiness is 19.6, agent-aware, with spec presence, auth clarity and rate limit signals lit and error semantics, idempotency and the MCP server dimension dark — the last despite an LiteLLM MCP API page in the record. A gateway that writes this honestly about the limits of its numbers deserves a catalog record that is equally precise about its surface, and the duplicate pages and the absent decisions route say the record has not caught up with the product.