Evals

name: Evals
description: A landscape catalog of the platforms, frameworks, libraries, and benchmark suites used to evaluate large language
  models, LLM-based applications, and AI agents. The topic spans human-rated, LLM-as-a-judge, reference-based, reference-free,
  and benchmark-aligned approaches to measuring AI system quality. Tracked alongside the eval platforms are the canonical
  multi-task and code/agent benchmark suites (MMLU, HumanEval, GAIA, AgentBench, BIG-Bench) that establish public points of
  comparison.
url: https://github.com/api-evangelist/evals
created: '2026-05-22'
modified: '2026-05-22'
specificationVersion: '0.18'
tags:
- Evals
- LLM Evaluation
- AI Quality
- Benchmarks
- LLM as a Judge
- Observability
- Agent Evaluation
- RAG Evaluation
- Test-Driven AI
apis:
- name: OpenAI Evals
  description: OpenAI Evals is the open-source framework released by OpenAI for evaluating large language models and LLM-based
    systems. The README states "Evals provide a framework for evaluating large language models (LLMs) or systems built using
    LLMs." The repo bundles a registry of benchmark evals, support for model-graded grading without writing custom code, private
    eval data via Snowflake logging, and templates for prompt chains and tool-using agents. Written primarily in Python, the
    project sits at roughly 18.5k stars / 3k forks.
  humanURL: https://github.com/openai/evals
  baseURL: https://github.com/openai/evals
  tags:
  - OpenAI
  - Open Source
  - Model Graded
  - Benchmark Registry
  - Python
  properties:
  - type: GitHubRepository
    url: https://github.com/openai/evals
  - type: Documentation
    url: https://github.com/openai/evals/tree/main/docs
  - type: License
    url: https://github.com/openai/evals/blob/main/LICENSE.md
- name: Inspect AI
  description: Inspect AI is an open-source framework for large language model evaluations developed and maintained by the
    UK AI Security Institute (UK AISI) and Meridian Labs. It supports text comparisons, model-based grading such as model_graded_fact(),
    and custom scorers. Datasets carry input and target columns, with multimodal support across image, audio, and video. The
    framework targets frontier-AI capability and safety assessment across coding, reasoning, knowledge, behavior, and multimodal
    understanding.
  humanURL: https://inspect.aisi.org.uk/
  baseURL: https://inspect.aisi.org.uk
  tags:
  - UK AISI
  - Open Source
  - Frontier AI
  - Model Graded
  - Safety Evaluation
  properties:
  - type: Documentation
    url: https://inspect.aisi.org.uk/
  - type: GitHubRepository
    url: https://github.com/UKGovernmentBEIS/inspect_ai
- name: Braintrust
  description: Braintrust is a commercial evaluation platform that captures eval runs as immutable, comparable experiment
    snapshots. The product supports code-based scorers, built-in autoevals, and LLM-as-a-judge evaluators for both offline
    and production use. Datasets are collections of test cases (input, optional expected output, metadata) sourced from production
    logs, user feedback, or manual curation. Experiments slot into CI/CD pipelines to detect regressions "before they reach
    production."
  humanURL: https://www.braintrust.dev/
  baseURL: https://www.braintrust.dev
  tags:
  - Commercial
  - LLM as a Judge
  - CI/CD
  - Experiments
  - Regression Detection
  properties:
  - type: Documentation
    url: https://www.braintrust.dev/docs
  - type: EvaluationGuide
    url: https://www.braintrust.dev/docs/guides/evals
  - type: Pricing
    url: https://www.braintrust.dev/pricing
- name: LangSmith Evaluation
  description: LangSmith Evaluation is LangChain's evaluation framework for measuring application quality across the lifecycle.
    The docs describe evals as "a way to breakdown what 'good' looks like and measure it." It supports code evaluators (deterministic
    rules), LLM-as-judge evaluators (reference-based or reference-free), and heuristic checks (length, latency, keywords).
    Concepts include datasets and examples, experiments, and pairwise evaluation for relative comparisons.
  humanURL: https://docs.langchain.com/langsmith/evaluation-concepts
  baseURL: https://api.smith.langchain.com
  tags:
  - LangChain
  - LLM as a Judge
  - Pairwise
  - Reference-Free
  - Online and Offline
  properties:
  - type: Documentation
    url: https://docs.langchain.com/langsmith/evaluation-concepts
  - type: Portal
    url: https://smith.langchain.com
  - type: Pricing
    url: https://www.langchain.com/pricing-langsmith
- name: Promptfoo
  description: Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. The docs describe
    it as enabling "test-driven LLM development rather than trial-and-error" and producing "matrix views that let you quickly
    evaluate outputs across many prompts." It supports assertion-based scoring, integrations across OpenAI, Anthropic, Azure,
    Google, HuggingFace, and open-source models, plus automated red-team and pentest runs that produce vulnerability and risk
    reports.
  humanURL: https://www.promptfoo.dev/
  baseURL: https://www.promptfoo.dev
  tags:
  - Open Source
  - CLI
  - Red Teaming
  - Assertions
  - Test-Driven
  properties:
  - type: Documentation
    url: https://www.promptfoo.dev/docs/intro/
  - type: GitHubRepository
    url: https://github.com/promptfoo/promptfoo
  - type: RedTeam
    url: https://www.promptfoo.dev/docs/red-team/
- name: Helicone
  description: Helicone is an open-source observability and monitoring platform for LLM applications. The homepage states
    "The world's fastest-growing AI companies rely on Helicone to route, debug, and analyze their applications." Beyond observability
    dashboards (requests, segments, sessions, users), it offers prompt management, datasets, a playground, rate-limit tracking,
    and alerts. LLM-as-a-judge style evaluation runs against captured request logs.
  humanURL: https://www.helicone.ai/
  baseURL: https://api.helicone.ai
  tags:
  - Open Source
  - Observability
  - Proxy
  - Prompt Management
  - Y Combinator
  properties:
  - type: Documentation
    url: https://docs.helicone.ai/
  - type: GitHubRepository
    url: https://github.com/Helicone/helicone
  - type: Pricing
    url: https://www.helicone.ai/pricing
- name: Patronus AI
  description: Patronus AI is a frontier lab building evaluation infrastructure and Digital World Models for human-aligned
    AGI. Its evaluator models include Lynx (a hallucination-detection model reported to outperform GPT-4 on hallucination
    tasks) and GLIDER (an evaluation model producing reasoning chains with explainable judgments). Coverage spans research
    science, software development, customer service, product applications, finance, and multi-turn dialogue / long-horizon
    task planning.
  humanURL: https://www.patronus.ai/
  baseURL: https://api.patronus.ai
  tags:
  - Commercial
  - Hallucination Detection
  - Judge Models
  - Lynx
  - GLIDER
  properties:
  - type: Documentation
    url: https://docs.patronus.ai/
  - type: Portal
    url: https://app.patronus.ai
- name: DeepEval (Confident AI)
  description: DeepEval is an open-source LLM evaluation package, paired with Confident AI as the hosted observability/evals/monitoring
    tier. The docs call DeepEval "an open-source LLM eval package" and Confident AI "an AI quality platform with observability,
    evals, and monitoring." Metrics include GEval (research-backed custom metric), AnswerRelevancyMetric, TaskCompletionMetric,
    and ConversationalGEval. Test cases use LLMTestCase and ConversationalTestCase shapes; datasets organize Golden test cases
    for sync or async runs.
  humanURL: https://www.deepeval.com/
  baseURL: https://api.confident-ai.com
  tags:
  - Open Source
  - GEval
  - RAG
  - Conversational
  - Python
  properties:
  - type: Documentation
    url: https://www.deepeval.com/docs/getting-started
  - type: GitHubRepository
    url: https://github.com/confident-ai/deepeval
  - type: Portal
    url: https://app.confident-ai.com
- name: Arize AI (Phoenix)
  description: Arize AI provides an AI observability and evaluation platform centered on Arize AX (the commercial product)
    and Phoenix (open-source LLM tracing and evaluation). Phoenix runs LLM-as-a-judge evaluators across traces, supports datasets
    and experiments, and integrates with OpenTelemetry. Arize AX layers monitoring, drift detection, and root-cause analysis
    on top of model and LLM telemetry.
  humanURL: https://arize.com/
  baseURL: https://api.arize.com
  tags:
  - Commercial
  - Open Source
  - Phoenix
  - OpenTelemetry
  - Observability
  properties:
  - type: Documentation
    url: https://arize.com/docs/ax/
  - type: GitHubRepository
    url: https://github.com/Arize-ai/phoenix
  - type: Portal
    url: https://app.arize.com
- name: Galileo
  description: Galileo is an enterprise AI observability and evaluation engineering platform. The product line emphasizes
    "20+ built-in evaluators" spanning RAG, agents, safety, and security, plus custom evaluators that "auto-tune metrics from
    live feedback." Luna refers to compact distilled evaluator models that "monitor 100% of your traffic at 97% lower cost."
    The homepage tagline reads "Don't just monitor AI failures. Stop them."
  humanURL: https://www.galileo.ai/
  baseURL: https://api.galileo.ai
  tags:
  - Commercial
  - Enterprise
  - Luna
  - RAG
  - Safety
  properties:
  - type: Documentation
    url: https://docs.galileo.ai/
  - type: Portal
    url: https://app.galileo.ai
- name: Humanloop
  description: Humanloop was a development platform for LLM applications, describing itself as having been "the first development
    platform for LLM applications" and having "shaped industry standards for how to manage and evaluate AI." Following its
    acquisition by Anthropic the platform has been sunset, with a migration path published for former customers. Retained
    in this catalog for historical completeness.
  humanURL: https://humanloop.com/
  baseURL: https://api.humanloop.com
  tags:
  - Historical
  - Acquired
  - Anthropic
  - Prompt Management
  properties:
  - type: Documentation
    url: https://humanloop.com/docs
  - type: AcquisitionNotice
    url: https://humanloop.com/
- name: TruLens
  description: TruLens is an open-source evaluation and tracing platform for AI agents that helps developers "move from vibes
    to metrics." Its feedback-function library covers the RAG triad — groundedness (responses supported by retrieved content),
    context relevance (retrieved documents match the query), and answer relevance (responses address the user question) —
    plus coherence, comprehensiveness, toxicity, sentiment, fairness, and custom metrics. Integrates with OpenTelemetry traces
    and any agent framework.
  humanURL: https://www.trulens.org/
  baseURL: https://www.trulens.org
  tags:
  - Open Source
  - RAG Triad
  - Feedback Functions
  - Snowflake
  - OpenTelemetry
  properties:
  - type: Documentation
    url: https://www.trulens.org/
  - type: GitHubRepository
    url: https://github.com/truera/trulens
- name: Weights and Biases Weave
  description: W&B Weave is a platform for evaluating, monitoring, and iterating on AI agents and applications, started with
    "one line of code." Weave Evaluations enable visual comparison of runs, automatic versioning of datasets and scorers,
    an interactive playground, and leaderboards. Scorers include pre-built ones (toxicity, hallucination), custom Python scoring
    functions, human feedback collection, and third-party scorers from providers such as RAGAS and LangChain.
  humanURL: https://wandb.ai/site/weave
  baseURL: https://api.wandb.ai
  tags:
  - Weights and Biases
  - Commercial
  - Scorers
  - Leaderboards
  - Human Feedback
  properties:
  - type: Documentation
    url: https://weave-docs.wandb.ai/
  - type: GitHubRepository
    url: https://github.com/wandb/weave
  - type: Portal
    url: https://wandb.ai
- name: Ragas
  description: Ragas is an open-source evaluation library focused on retrieval-augmented generation, described in its own
    docs as "a library that helps you move from 'vibe checks' to systematic evaluation loops for your AI applications." It
    exposes LLM-driven metrics for RAG (faithfulness, context recall, answer relevancy), integrates with LangChain and LlamaIndex,
    and supports custom metric authoring as a complement to other eval platforms (Weave, LangSmith).
  humanURL: https://docs.ragas.io/
  baseURL: https://docs.ragas.io
  tags:
  - Open Source
  - RAG
  - Faithfulness
  - Context Recall
  - Library
  properties:
  - type: Documentation
    url: https://docs.ragas.io/en/stable/
  - type: GitHubRepository
    url: https://github.com/explodinggradients/ragas
- name: MLflow LLM Evaluate
  description: MLflow LLM evaluate extends MLflow's experiment tracking with mlflow.evaluate() support for LLM tasks. The
    API runs reference-based and reference-free metrics (toxicity, perplexity, BLEU, ROUGE, exact match, custom LLM judges)
    over a logged model or a function and persists results into MLflow's experiment store alongside traditional ML metrics.
    Sits inside the broader MLflow open-source project.
  humanURL: https://mlflow.org/
  baseURL: https://mlflow.org
  tags:
  - Open Source
  - MLflow
  - Experiment Tracking
  - LLM Judges
  - Apache
  properties:
  - type: Documentation
    url: https://mlflow.org/docs/latest/llms/llm-evaluate/index.html
  - type: GitHubRepository
    url: https://github.com/mlflow/mlflow
- name: MMLU Benchmark
  description: MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 subjects
    from STEM and international law to nutrition and religion. It contains 15,908 multiple-choice questions (four options
    each), of which 1,540 are reserved for hyperparameter tuning. Per its overview, "It was one of the most commonly used
    benchmarks for comparing the capabilities of large language models, with over 100 million downloads as of July 2024."
  humanURL: https://github.com/hendrycks/test
  baseURL: https://github.com/hendrycks/test
  tags:
  - Benchmark
  - Knowledge
  - Multiple Choice
  - Multitask
  - Reference-Based
  properties:
  - type: GitHubRepository
    url: https://github.com/hendrycks/test
  - type: Paper
    url: https://arxiv.org/abs/2009.03300
  - type: Dataset
    url: https://huggingface.co/datasets/cais/mmlu
- name: HumanEval Benchmark
  description: HumanEval is OpenAI's evaluation harness for code-generation models, described in its README as "an evaluation
    harness for the HumanEval problem solving dataset described in the paper 'Evaluating Large Language Models Trained on
    Code'." Functional correctness is measured by executing model-generated code against unit tests, reported as pass@1, pass@10,
    and pass@100 by default.
  humanURL: https://github.com/openai/human-eval
  baseURL: https://github.com/openai/human-eval
  tags:
  - Benchmark
  - Code Generation
  - Functional Correctness
  - Pass@k
  - Reference-Based
  properties:
  - type: GitHubRepository
    url: https://github.com/openai/human-eval
  - type: Paper
    url: https://arxiv.org/abs/2107.03374
  - type: Dataset
    url: https://huggingface.co/datasets/openai/openai_humaneval
- name: GAIA Benchmark
  description: GAIA is "a benchmark for General AI Assistants," published in 2023 (arXiv 2311.12983). It tests general-purpose
    AI agent capability across reasoning, tool use, multi-modality, and web browsing, with a public leaderboard hosted on
    Hugging Face for community submissions. The benchmark has become a reference point for evaluating agentic systems that
    combine an LLM with tools and a browser.
  humanURL: https://huggingface.co/gaia-benchmark
  baseURL: https://huggingface.co/gaia-benchmark
  tags:
  - Benchmark
  - AI Agents
  - Reasoning
  - Tool Use
  - Leaderboard
  properties:
  - type: Dataset
    url: https://huggingface.co/datasets/gaia-benchmark/GAIA
  - type: Paper
    url: https://arxiv.org/abs/2311.12983
  - type: Leaderboard
    url: https://huggingface.co/spaces/gaia-benchmark/leaderboard
- name: AgentBench
  description: AgentBench is the first benchmark designed to evaluate LLM-as-Agent across a diverse spectrum of environments.
    It bundles 8 environments — 5 newly created (Operating System, Database, Knowledge Graph, Digital Card Game, Lateral Thinking
    Puzzles) and 3 adapted (House-Holding from ALFWorld, Web Shopping from WebShop, Web Browsing from Mind2Web). The benchmark
    requires roughly 4,000 dev-set and 13,000 test-set interactions per model.
  humanURL: https://github.com/THUDM/AgentBench
  baseURL: https://github.com/THUDM/AgentBench
  tags:
  - Benchmark
  - AI Agents
  - Multi-Environment
  - LLM-as-Agent
  - Tsinghua
  properties:
  - type: GitHubRepository
    url: https://github.com/THUDM/AgentBench
  - type: Paper
    url: https://arxiv.org/abs/2308.03688
  - type: Leaderboard
    url: https://llmbench.ai/agent
- name: BIG-Bench
  description: The Beyond the Imitation Game Benchmark (BIG-Bench) is "a collaborative benchmark intended to probe large language
    models and extrapolate their future capabilities." It contains more than 200 tasks across JSON-based simplified tasks
    and programmatic tasks; a curated subset (BIG-Bench Lite) of 24 tasks is provided as the canonical headline measurement.
    Maintained on GitHub by Google with open community task submissions.
  humanURL: https://github.com/google/BIG-bench
  baseURL: https://github.com/google/BIG-bench
  tags:
  - Benchmark
  - Collaborative
  - Multitask
  - Google
  - BIG-Bench Lite
  properties:
  - type: GitHubRepository
    url: https://github.com/google/BIG-bench
  - type: Paper
    url: https://arxiv.org/abs/2206.04615
  - type: Documentation
    url: https://github.com/google/BIG-bench/blob/main/README.md
common:
- type: GitHubOrganization
  url: https://github.com/api-evangelist
- type: JSONSchema
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-schema/evals-eval-run-schema.json
  title: Eval Run Schema
- type: JSONSchema
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-schema/evals-eval-suite-schema.json
  title: Eval Suite Schema
- type: JSONSchema
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-schema/evals-eval-case-schema.json
  title: Eval Case Schema
- type: JSONSchema
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-schema/evals-scorer-schema.json
  title: Scorer Schema
- type: JSONSchema
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-schema/evals-judge-schema.json
  title: Judge Schema
- type: JSONSchema
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-schema/evals-dataset-schema.json
  title: Dataset Schema
- type: JSONStructure
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-structure/evals-eval-run-structure.json
  title: Eval Run Structure
- type: JSONLD
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-ld/evals-context.jsonld
- type: Vocabulary
  url: https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/vocabulary/evals-vocabulary.yml
- type: Features
  data:
  - name: LLM-as-a-Judge Scoring
    description: A second LLM evaluates the output of the system-under-test, producing a numeric or categorical score and
      (optionally) a written rationale. The dominant scoring mode for free-form text outputs across Braintrust, LangSmith,
      DeepEval, Weave, TruLens, Phoenix, and Patronus.
  - name: Reference-Based Scoring
    description: Compares model output against a ground-truth expected answer using exact match, BLEU, ROUGE, embedding similarity,
      or task-specific equality (e.g. unit-test pass/fail). The native mode for benchmarks like MMLU, HumanEval, and GAIA.
  - name: Reference-Free Scoring
    description: Assesses output quality without ground truth — toxicity, coherence, faithfulness against retrieved context,
      criterion adherence. Enables online (production-traffic) evaluation where labels do not exist.
  - name: Pairwise Comparison
    description: A judge ranks two candidate outputs A vs B (or a tie). Useful when absolute scoring is hard but relative
      preference is reliable. Surfaced explicitly by LangSmith and used widely in chatbot arenas.
  - name: Benchmark-Aligned Evaluation
    description: Runs the system-under-test against a standardized public dataset (MMLU, HumanEval, GAIA, AgentBench, BIG-Bench)
      to produce comparable, headline scores. The basis of model leaderboards.
  - name: Human-Rated Scoring
    description: Domain experts or end users provide thumbs-up/down, Likert ratings, or written critiques. Used as ground
      truth, as a judge-calibration signal, and as a final acceptance gate before production.
  - name: RAG Triad
    description: Three feedback functions — groundedness, context relevance, answer relevance — codified by TruLens and widely
      adopted across Ragas, Phoenix, DeepEval, and LangSmith for evaluating retrieval-augmented generation pipelines.
  - name: Agent and Tool-Use Evaluation
    description: Evaluating multi-step agent trajectories — did the agent pick the right tool, did the tool call succeed,
      did the final answer satisfy the goal. Supported by Inspect AI, Galileo, Weave, LangSmith, Braintrust, and benchmarks
      like AgentBench and GAIA.
  - name: Online Production Monitoring
    description: Eval scorers (typically reference-free LLM judges and Luna-style distilled evaluators) attach to live traffic
      via tracing/observability layers (Phoenix, Arize, Helicone, Galileo, Weave) to flag regressions in real time.
  - name: Red-Team / Safety Evaluation
    description: Adversarial test suites probe jailbreaks, prompt injection, PII leakage, harmful content, and policy violations.
      First-class in Promptfoo, Patronus, Galileo, and Inspect AI's safety evals.
- type: UseCases
  data:
  - name: Model Selection
    description: Run candidate models (GPT-5, Claude 4.7, Gemini 3, open-weight) against a shared eval suite to choose the
      best fit for a specific application by quality, cost, and latency.
  - name: Prompt Engineering Iteration
    description: Compare prompt variants in a matrix-style eval (Promptfoo, LangSmith experiments, Braintrust experiments)
      to pick the best prompt before shipping.
  - name: Regression Detection in CI/CD
    description: Wire an eval suite into CI so a pull request that drops a key scorer below a threshold fails the build, preventing
      quality regressions from reaching production.
  - name: RAG Pipeline Tuning
    description: Use RAG-triad scores (groundedness / context relevance / answer relevance) and faithfulness to tune chunking,
      embedding, reranking, and prompt choices.
  - name: Agent Trajectory Quality
    description: Score multi-step agent runs on tool-selection correctness, step efficiency, and final-answer faithfulness
      — the core measurement for production agentic apps.
  - name: Hallucination and Safety Guardrails
    description: Deploy dedicated judge models (Lynx, GLIDER, Luna) to flag hallucinations, toxic content, PII leakage, and
      policy violations in real time.
  - name: Frontier Capability and Safety Assessment
    description: Independent labs (UK AISI, US AISI) run capability and safety evaluations on frontier models before release,
      using frameworks like Inspect AI.
  - name: Public Leaderboard Reporting
    description: Submit a model's scores against MMLU, HumanEval, GAIA, AgentBench, and BIG-Bench to position it on community
      leaderboards and back marketing claims with reproducible numbers.
- type: Integrations
  data:
  - name: OpenTelemetry
    description: Phoenix, TruLens, Weave, and most modern eval platforms ingest LLM traces via OpenTelemetry, making eval
      a layer on top of standard observability.
  - name: LangChain / LangGraph
    description: LangSmith is the native eval tier for LangChain/LangGraph apps; most other platforms also integrate.
  - name: LlamaIndex
    description: Ragas, DeepEval, and Phoenix integrate directly with LlamaIndex for RAG evaluation.
  - name: Hugging Face Datasets
    description: MMLU, HumanEval, and GAIA are distributed as Hugging Face datasets and consumed by every eval framework.
  - name: CI/CD (GitHub Actions, etc.)
    description: Braintrust, Promptfoo, LangSmith, and DeepEval ship CI integrations to fail builds on regression.
  - name: Snowflake
    description: OpenAI Evals can log eval results to Snowflake; TruLens (Truera) is now part of Snowflake.
maintainers:
- FN: Kin Lane
  email: info@apievangelist.com