Ragas is an open-source evaluation toolkit for Large Language Model applications, with particular depth on Retrieval Augmented Generation (RAG) and agentic systems. Originally created under the Exploding Gradients organization on GitHub and now maintained by Vibrant Labs AI, Ragas is a Python library distributed on PyPI under the Apache 2.0 license. It moves teams from informal "vibe checks" to systematic evaluation loops by providing objective LLM-based and traditional metrics, automated test dataset generation, experiment tracking, and integrations with the broader LLM ecosystem including LangChain, LlamaIndex, OpenAI, Anthropic, and popular observability platforms. Ragas exposes a metrics library covering faithfulness, response relevancy, context precision and recall, factual correctness, semantic similarity, agent tool-use accuracy, SQL equivalence, Nvidia-defined RAG metrics, and general-purpose rubric scoring. The project ships a CLI (`ragas`) with quickstart templates such as `rag_eval`, and is consumed primarily as a `pip install ragas` library rather than as a hosted API service. Ragas is widely cited as a default evaluation harness for RAG applications and has grown a substantial community on GitHub and Discord.
Ragas publishes 1 API on the APIs.io network. Tagged areas include LLM Evaluation, RAG Evaluation, Retrieval Augmented Generation, AI Evaluation, and Open-Source.
Ragas’ developer surface includes documentation, getting-started guide, release notes, and 17 more developer resources.
Create-or-Update Ergonomics could not be measured. We hold no machine-readable contract for
this provider to read, so there is nothing to measure a write surface against. Excluded rather than scored zero:
never-measured and measured-empty are different facts. Publishing an OpenAPI is what makes this facet — and
several others — scorable at all.
Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from github.com/api-evangelist/ragas-ai: open an issue to ask a question, or submit a pull request to add artifacts.
Submit an artifact on GitHub — free →Manage your own listing — the Influence plan, $499/mo →
The Ragas Python library is the primary surface of the project, installed via `pip install ragas` and imported as `ragas`. It exposes evaluation entry points (`ragas.evaluate`),...
aid: ragas-ai
name: Ragas
description: Ragas is an open-source evaluation toolkit for Large Language Model applications, with particular depth on Retrieval
Augmented Generation (RAG) and agentic systems. Originally created under the Exploding Gradients organization on GitHub
and now maintained by Vibrant Labs AI, Ragas is a Python library distributed on PyPI under the Apache 2.0 license. It moves
teams from informal "vibe checks" to systematic evaluation loops by providing objective LLM-based and traditional metrics,
automated test dataset generation, experiment tracking, and integrations with the broader LLM ecosystem including LangChain,
LlamaIndex, OpenAI, Anthropic, and popular observability platforms. Ragas exposes a metrics library covering faithfulness,
response relevancy, context precision and recall, factual correctness, semantic similarity, agent tool-use accuracy, SQL
equivalence, Nvidia-defined RAG metrics, and general-purpose rubric scoring. The project ships a CLI (`ragas`) with quickstart
templates such as `rag_eval`, and is consumed primarily as a `pip install ragas` library rather than as a hosted API service.
Ragas is widely cited as a default evaluation harness for RAG applications and has grown a substantial community on GitHub
and Discord.
type: Index
deliveryModel:
model: unknown
open_source: unknown
commercial: false
callable_host: false
label: Delivery model not determined — needs a product licence on record
confidence: low
source:
- repository-unlicensed
generated: '2026-08-28'
method: derived
accessModel:
pricing: unknown
onboarding: unknown
trial: false
try_now: false
public: false
label: Unknown
confidence: low
source: []
generated: '2026-07-22'
method: derived
position: Producing
access: 3rd-Party
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/ragas-ai.png
tags:
- LLM Evaluation
- RAG Evaluation
- Retrieval Augmented Generation
- AI Evaluation
- Open-Source
- Python
- Metrics
- Test Data Generation
- Agent Evaluation
- LLM Tooling
tags_raw:
- LLM Evaluation
- RAG Evaluation
- Retrieval Augmented Generation
- AI Evaluation
- Open Source
- Python
- Metrics
- Test Data Generation
- Agent Evaluation
- LLM Tooling
url: https://raw.githubusercontent.com/api-evangelist/ragas-ai/refs/heads/main/apis.yml
created: '2026-05-25'
modified: '2026-05-25'
specificationVersion: '0.23'
apis:
- aid: ragas-ai:ragas
name: Ragas Python Library
description: The Ragas Python library is the primary surface of the project, installed via `pip install ragas` and imported
as `ragas`. It exposes evaluation entry points (`ragas.evaluate`), metric classes (Faithfulness, AnswerRelevancy, ContextPrecision,
ContextRecall, FactualCorrectness, SemanticSimilarity, ToolCallAccuracy, AgentGoalAccuracy, and more), dataset generation
utilities, and integrations with LangChain and LlamaIndex. The library is not an HTTP API — it is consumed in-process
by Python evaluation scripts, notebooks, and CI pipelines.
humanURL: https://docs.ragas.io/
tags:
- Python
- Library
- Evaluation
- RAG
properties:
- url: https://docs.ragas.io/
type: Documentation
- url: https://github.com/explodinggradients/ragas
type: SourceCode
- url: https://pypi.org/project/ragas/
type: SDKs
- url: https://github.com/explodinggradients/ragas/blob/main/LICENSE
type: License
common:
- type: SecurityPolicy
url: https://github.com/vibrantlabsai/ragas/blob/main/SECURITY.md
- type: CodeOfConduct
url: https://github.com/vibrantlabsai/ragas/blob/main/CODE_OF_CONDUCT.md
- type: ContributionGuide
url: https://github.com/vibrantlabsai/ragas/blob/main/CONTRIBUTING.md
- type: DomainSecurity
url: security/ragas-ai-domain-security.yml
- type: Website
url: https://www.ragas.io/
- type: Documentation
url: https://docs.ragas.io/
- type: GettingStarted
url: https://docs.ragas.io/en/stable/getstarted/
- type: Concepts
url: https://docs.ragas.io/en/stable/concepts/
- type: Metrics
url: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/
- type: HowToGuides
url: https://docs.ragas.io/en/stable/howtos/
- type: SourceCode
url: https://github.com/explodinggradients/ragas
- type: GitHubOrganization
url: https://github.com/explodinggradients
- type: Package
url: https://pypi.org/project/ragas/
- type: License
url: https://github.com/explodinggradients/ragas/blob/main/LICENSE
- type: Issues
url: https://github.com/explodinggradients/ragas/issues
- type: ReleaseNotes
url: https://github.com/explodinggradients/ragas/releases
- type: Discord
url: https://discord.gg/5djav8GGNZ
- type: Twitter
url: https://twitter.com/ragas_io
- type: Company
url: https://www.vibrantlabs.ai/
- type: Contact
url: mailto:founders@vibrantlabs.com
- type: Features
data:
- name: RAG Evaluation Metrics
description: Faithfulness, Response Relevancy, Context Precision, Context Recall, Context Entities Recall, and Noise Sensitivity
for retrieval augmented generation pipelines.
- name: Agent and Tool-Use Metrics
description: Topic Adherence, Tool Call Accuracy, Tool Call F1, and Agent Goal Accuracy for evaluating multi-step agentic
systems.
- name: Natural Language Comparison
description: Factual Correctness, Semantic Similarity, BLEU, ROUGE, CHRF, Exact Match, and String Presence metrics for
output comparison.
- name: SQL Evaluation
description: Execution-based Datacompy Score and SQL Query Equivalence metrics for text-to-SQL applications.
- name: General Purpose Scoring
description: Aspect Critic, Simple Criteria Scoring, Rubrics-based scoring, and instance-specific rubrics for custom evaluation
criteria.
- name: Nvidia Metrics
description: Answer Accuracy, Context Relevance, and Response Groundedness metrics contributed by Nvidia for RAG quality.
- name: Test Data Generation
description: Automated synthesis of diverse test datasets covering single-hop, multi-hop, and abstract query types over
user knowledge bases.
- name: Experiments
description: Experiment-first workflow comparing prompts, models, and configurations across datasets with iterative result
tracking.
- name: Custom Metrics
description: DiscreteMetric and decorator-based APIs for defining LLM-judge and rule-based custom evaluation metrics.
- name: CLI Quickstart Templates
description: The `ragas quickstart` command scaffolds evaluation projects including the `rag_eval` template for RAG systems.
- type: UseCases
data:
- name: RAG Pipeline Evaluation
description: Scoring retrieval and generation quality in RAG applications across faithfulness, relevance, and context
fidelity.
- name: Agent Evaluation
description: Measuring tool-call correctness, goal completion, and topic adherence in multi-step LLM agents.
- name: Regression Testing in CI
description: Running Ragas metrics in CI pipelines to detect quality regressions across prompt, model, and configuration
changes.
- name: Model and Prompt Selection
description: Comparing candidate models and prompt variants on a fixed dataset using Ragas experiments.
- name: Synthetic Test Set Generation
description: Generating diverse evaluation datasets from a knowledge base for systematic LLM testing.
- name: Text-to-SQL Evaluation
description: Validating generated SQL against reference queries using execution and structural equivalence metrics.
- type: Integrations
data:
- name: LangChain
description: Native integration for evaluating LangChain chains, retrievers, and agents using Ragas metrics.
- name: LlamaIndex
description: Integration for evaluating LlamaIndex RAG pipelines and query engines.
- name: OpenAI
description: Default LLM judge backend uses OpenAI models such as GPT-4 class judges.
- name: Anthropic
description: Anthropic Claude models supported as LLM judges via the LangChain LLM abstraction.
- name: Hugging Face
description: Support for Hugging Face embeddings and models as judges, plus dataset interop via the `datasets` library.
- name: LangSmith
description: Result tracking and trace inspection via LangSmith observability.
- name: Arize Phoenix
description: Observability integration for tracing Ragas evaluations alongside production LLM traffic.
- name: Helicone
description: LLM cost and trace observability for Ragas-driven evaluations.
- name: Pandas
description: Datasets and evaluation results are exposed as pandas DataFrames for analysis.
maintainers:
- FN: Kin Lane
email: kin@apievangelist.com
Discovery needs no key. Ratings and market analysis are Pro.
Get an API key
Free tier, no form to fill in. Signing in shares your email address with us — we
store it to create your key and to recognise you if you sign in with another
provider. See our Privacy Policy and
Terms.
Kin Score
How this is scored →