# Evals **Canonical:** https://apis.io/providers/evals/ **APIs profiled:** 20 A landscape catalog of the platforms, frameworks, libraries, and benchmark suites used to evaluate large language models, LLM-based applications, and AI agents. The topic spans human-rated, LLM-as-a-judge, reference-based, reference-free, and benchmark-aligned approaches to measuring AI system quality. Tracked alongside the eval platforms are the canonical multi-task and code/agent benchmark suites (MMLU, HumanEval, GAIA, AgentBench, BIG-Bench) that establish public points of comparison. ## Kin Score — 16.0 / 100 (emerging) Scored 2026-08-20 under rubric 0.12.0. Trend: flat (+0.0 from 16.0). | Facet | Score | |---|---| | Discoverability | 74.1 | | Contract Quality | 15.5 | | Governance | 25.0 | | Contract Governance | 25.0 | | Operational Transparency | 13.2 | | Developer Ergonomics | 0.0 | | Commercial Clarity | 0.0 | | Access Clarity | 0.0 | ## Agent readiness — 0.0 (human-only) | Dimension | Value | |---|---| | Spec Presence | no | | Agentic Access | no | | Reversibility Documented | no | | MCP Server | no | | Auth Clarity | no | | Idempotency | no | | Error Semantics | no | | OpenAPI Examples | no | | Rate Limit Signal | no | | Event Surface Described | no | | Agent Skills | no | | Well Known Catalog | no | | Consent Identity | no | | Agent Card | no | | Dry Run Mode | no | ## Access Unknown — onboarding: unknown, pricing: unknown, trial: no (confidence: low). ## APIs (20) - **OpenAI Evals** — OpenAI Evals is the open-source framework released by OpenAI for evaluating large language models and LLM-based systems. The README states "Evals provide a framework for evaluat... - **Inspect AI** — Inspect AI is an open-source framework for large language model evaluations developed and maintained by the UK AI Security Institute (UK AISI) and Meridian Labs. It supports tex... - **Braintrust** — Braintrust is a commercial evaluation platform that captures eval runs as immutable, comparable experiment snapshots. The product supports code-based scorers, built-in autoevals... - **LangSmith Evaluation** — LangSmith Evaluation is LangChain's evaluation framework for measuring application quality across the lifecycle. The docs describe evals as "a way to breakdown what 'good' looks... - **Promptfoo** — Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. The docs describe it as enabling "test-driven LLM development rather than trial-and-... - **Helicone** — Helicone is an open-source observability and monitoring platform for LLM applications. The homepage states "The world's fastest-growing AI companies rely on Helicone to route, d... - **Patronus AI** — Patronus AI is a frontier lab building evaluation infrastructure and Digital World Models for human-aligned AGI. Its evaluator models include Lynx (a hallucination-detection mod... - **DeepEval (Confident AI)** — DeepEval is an open-source LLM evaluation package, paired with Confident AI as the hosted observability/evals/monitoring tier. The docs call DeepEval "an open-source LLM eval pa... - **Arize AI (Phoenix)** — Arize AI provides an AI observability and evaluation platform centered on Arize AX (the commercial product) and Phoenix (open-source LLM tracing and evaluation). Phoenix runs LL... - **Galileo** — Galileo is an enterprise AI observability and evaluation engineering platform. The product line emphasizes "20+ built-in evaluators" spanning RAG, agents, safety, and security, ... - **Humanloop** — Humanloop was a development platform for LLM applications, describing itself as having been "the first development platform for LLM applications" and having "shaped industry sta... - **TruLens** — TruLens is an open-source evaluation and tracing platform for AI agents that helps developers "move from vibes to metrics." Its feedback-function library covers the RAG triad — ... - **Weights and Biases Weave** — W&B Weave is a platform for evaluating, monitoring, and iterating on AI agents and applications, started with "one line of code." Weave Evaluations enable visual comparison of r... - **Ragas** — Ragas is an open-source evaluation library focused on retrieval-augmented generation, described in its own docs as "a library that helps you move from 'vibe checks' to systemati... - **MLflow LLM Evaluate** — MLflow LLM evaluate extends MLflow's experiment tracking with mlflow.evaluate() support for LLM tasks. The API runs reference-based and reference-free metrics (toxicity, perplex... - **MMLU Benchmark** — MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 subjects from STEM and international law to nutrition and religion. It conta... - **HumanEval Benchmark** — HumanEval is OpenAI's evaluation harness for code-generation models, described in its README as "an evaluation harness for the HumanEval problem solving dataset described in the... - **GAIA Benchmark** — GAIA is "a benchmark for General AI Assistants," published in 2023 (arXiv 2311.12983). It tests general-purpose AI agent capability across reasoning, tool use, multi-modality, a... - **AgentBench** — AgentBench is the first benchmark designed to evaluate LLM-as-Agent across a diverse spectrum of environments. It bundles 8 environments — 5 newly created (Operating System, Dat... - **BIG-Bench** — The Beyond the Imitation Game Benchmark (BIG-Bench) is "a collaborative benchmark intended to probe large language models and extrapolate their future capabilities." It contains... ## Security (1) - **Evals Domain Security** — TLSv1.3 · HSTS · DMARC ## Use cases (8) - **Model Selection** — Run candidate models (GPT-5, Claude 4.7, Gemini 3, open-weight) against a shared eval suite to choose the best fit for a specific application by quality, cost, and latency. - **Prompt Engineering Iteration** — Compare prompt variants in a matrix-style eval (Promptfoo, LangSmith experiments, Braintrust experiments) to pick the best prompt before shipping. - **Regression Detection in CI/CD** — Wire an eval suite into CI so a pull request that drops a key scorer below a threshold fails the build, preventing quality regressions from reaching production. - **RAG Pipeline Tuning** — Use RAG-triad scores (groundedness / context relevance / answer relevance) and faithfulness to tune chunking, embedding, reranking, and prompt choices. - **Agent Trajectory Quality** — Score multi-step agent runs on tool-selection correctness, step efficiency, and final-answer faithfulness — the core measurement for production agentic apps. - **Hallucination and Safety Guardrails** — Deploy dedicated judge models (Lynx, GLIDER, Luna) to flag hallucinations, toxic content, PII leakage, and policy violations in real time. - **Frontier Capability and Safety Assessment** — Independent labs (UK AISI, US AISI) run capability and safety evaluations on frontier models before release, using frameworks like Inspect AI. - **Public Leaderboard Reporting** — Submit a model's scores against MMLU, HumanEval, GAIA, AgentBench, and BIG-Bench to position it on community leaderboards and back marketing claims with reproducible numbers. ## Tags Evals, LLM Evaluation, AI Quality, Benchmarks, LLM as a Judge, Observability, Agent Evaluation, RAG Evaluation, Test-Driven AI --- Profiled by [API Evangelist](https://apievangelist.com) and published on [APIs.io](https://apis.io/providers/evals/). Scores are computed from the provider's own public artifacts under a published rubric.