Relevance AI

Relevance AI Evals API

The evals API from Relevance AI — 22 operation(s) for evals.

GitHub OpenAPI
Business capability
Software Quality Engineering BC-4200.50

Operations 37

Each operation below carries the questions people ask an LLM about it and the instructions they give an agent to run it. Generated by API Evangelist overlay

POST /evals/{resource_type}/{resource_id}/test-cases Create a test case with a scenario and checks · CreateTestCase #
Ask an LLM
“How do I write a new eval test case describing a scenario for my agent?”
“Can a new test case be added straight into a specific test set?”
Tell an agent
Create test case {name} for {resource_type} {resource_id} with scenario {scenario} and checks {check_ids}.
Add a test case named {name} to test set {test_set_id} on agent {resource_id}.
GET /evals/{resource_type}/{resource_id}/test-cases List test cases for an agent or workforce · ListTestCases #
Ask an LLM
“What test cases have I written for my agent?”
“Can I list only the test cases in one test set?”
Tell an agent
List test cases for {resource_type} {resource_id}.
List test cases in test set {test_set_id} for {resource_type} {resource_id}.
GET /evals/{resource_type}/{resource_id}/batches List eval batches for an agent or workforce · ListEvalBatchByResourceNew #
Ask an LLM
“Which evaluation batches have been run against my agent?”
“Can I sort an agent's eval batches by a field and direction?”
Tell an agent
List eval batches for {resource_type} {resource_id}.
Search eval batches on {resource_type} {resource_id} for {search}.
POST /evals/{resource_type}/{resource_id}/rules Create an evaluator rule · CreateEvalRuleForResource #
Ask an LLM
“How do I add a new evaluator rule that grades my agent's conversations?”
“Can I define my own evaluation criteria for a workforce?”
Tell an agent
Create an evaluator rule on {resource_type} {resource_id} defined as {user_definition}.
Add a new eval rule to agent {resource_id}: {user_definition}.
GET /evals/{resource_type}/{resource_id}/rules List evaluator rules for an agent or workforce · ListEvalRulesForResource #
Ask an LLM
“What evaluator rules are defined for my workforce?”
“Which grading rules is my agent being evaluated against?”
Tell an agent
List the evaluator rules for {resource_type} {resource_id}.
Show all eval rules configured on agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/runs List eval runs for an agent or workforce · ListEvalRunsForResource #
Ask an LLM
“What individual eval runs have been done on my agent recently?”
“Can I filter eval runs by a date range?”
Tell an agent
List eval runs for {resource_type} {resource_id}.
Show eval runs on {resource_type} {resource_id} within {date_range}.
POST /evals/{resource_type}/{resource_id}/evaluate Run an evaluation on an agent or workforce · EvaluateResource #
Ask an LLM
“How do I evaluate a set of my agent's past conversations?”
“Can I run an evaluation using a whole test set and selected rules?”
Tell an agent
Start an evaluation run named {evaluation_run_name} on {resource_type} {resource_id} for conversations {conversation_ids}.
Evaluate agent {resource_id} against test set {test_set_id} as run {evaluation_run_name}.
GET /evals/{resource_type}/{resource_id}/batches/{eval_batch_id}/summary Get the summary of an eval batch · GetEvalBatchSummaryForResource #
Ask an LLM
“How did my agent score in a particular eval batch?”
“Can I get summary results for one evaluation batch?”
Tell an agent
Summarize eval batch {eval_batch_id} for {resource_type} {resource_id}.
Show results summary for batch {eval_batch_id} on agent {resource_id}.
DELETE /evals/{resource_type}/{resource_id}/batches/{eval_batch_id} Delete an eval batch · DeleteEvalBatchForResource #
Ask an LLM
“Can I delete an old evaluation batch I no longer need?”
“What removes an eval batch from an agent's history?”
Tell an agent destructive · confirm first
Delete eval batch {eval_batch_id} from {resource_type} {resource_id}.
Remove batch {eval_batch_id} from agent {resource_id}'s evals.
POST /evals/{resource_type}/{resource_id}/batches/{eval_batch_id}/cancel Cancel all pending runs in an eval batch · CancelEvalBatchForResource #
Ask an LLM
“How do I stop an entire eval batch that's still running?”
“Does cancelling a batch stop both pending and running eval runs?”
Tell an agent destructive · confirm first
Cancel eval batch {eval_batch_id} on {resource_type} {resource_id}.
Stop every pending run in batch {eval_batch_id} for agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/runs/{eval_run_id}/cancel Cancel a single eval run · CancelEvalRunForResource #
Ask an LLM
“Can I cancel just one eval run without stopping the rest of the batch?”
“What stops a single stuck evaluation run?”
Tell an agent destructive · confirm first
Cancel eval run {eval_run_id} on {resource_type} {resource_id}.
Stop only run {eval_run_id} for agent {resource_id}.
PATCH /evals/{resource_type}/{resource_id}/rules/{eval_rule_id} Update an evaluator rule · UpdateEvalRuleForResource #
Ask an LLM
“Can I change the definition of an existing evaluator rule?”
“How would I tweak the criteria of a rule already grading my agent?”
Tell an agent
Update evaluator rule {eval_rule_id} on {resource_type} {resource_id} to {user_definition}.
Change the definition of eval rule {eval_rule_id} for agent {resource_id}.
DELETE /evals/{resource_type}/{resource_id}/rules/{eval_rule_id} Delete an evaluator rule · DeleteEvalRuleForResource #
Ask an LLM
“How do I remove an evaluator rule from my agent?”
“Can I delete a grading rule that's no longer relevant?”
Tell an agent destructive · confirm first
Delete evaluator rule {eval_rule_id} from {resource_type} {resource_id}.
Remove eval rule {eval_rule_id} on agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/observability/config Create the legacy observability config · CreateObservabilityConfig #
Ask an LLM
“How was the single observability config created before multiple configs were supported?”
“Can I still create an observability config via the deprecated endpoint?”
Tell an agent
Create the legacy single observability config on {resource_type} {resource_id} with rules {rule_ids}.
Use the deprecated endpoint to set up observability on agent {resource_id} using rules {rule_ids}.
PUT /evals/{resource_type}/{resource_id}/observability/config Update the legacy observability config · UpdateObservabilityConfig #
Ask an LLM
“Can I still update an agent's observability settings through the older single-config endpoint?”
“What does the deprecated observability config update change?”
Tell an agent
Update the legacy observability config on {resource_type} {resource_id} to use rules {rule_ids}.
Set the deprecated observability config sample rate to {sample_rate} for agent {resource_id} with rules {rule_ids}.
GET /evals/{resource_type}/{resource_id}/observability/config Get the legacy observability config · GetObservabilityConfig #
Ask an LLM
“Where is the older single observability config for an agent read from?”
“Is the deprecated get-observability-config endpoint still usable?”
Tell an agent
Get the legacy single observability config for {resource_type} {resource_id}.
Read the deprecated observability config on agent {resource_id}.
DELETE /evals/{resource_type}/{resource_id}/observability/config Delete the legacy observability config · DeleteObservabilityConfig #
Ask an LLM
“Can I remove an agent's single observability config through the old endpoint?”
“What deletes the deprecated one-per-resource observability config?”
Tell an agent destructive · confirm first
Delete the legacy observability config on {resource_type} {resource_id}.
Remove the deprecated observability config from agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/observability/configs Add an observability config · CreateObservabilityConfigV2 #
Ask an LLM
“How do I add another observability config so live conversations get evaluated?”
“Can I sample only a percentage of conversations for ongoing evaluation?”
Tell an agent
Add an observability config named {name} to {resource_type} {resource_id} with rules {rule_ids}.
Create a new monitoring config on agent {resource_id} sampling {sample_rate} with rules {rule_ids}.
GET /evals/{resource_type}/{resource_id}/observability/configs List observability configs for an agent or workforce · ListObservabilityConfigs #
Ask an LLM
“Which observability configurations are set up on my workforce?”
“How many monitoring configs does my agent currently have?”
Tell an agent
List all observability configs for {resource_type} {resource_id}.
Show every monitoring config on agent {resource_id}.
PUT /evals/{resource_type}/{resource_id}/observability/configs/{config_id} Update a specific observability config · UpdateObservabilityConfigV2 #
Ask an LLM
“Can I change the rules or sample rate on one of several observability configs?”
“How do I disable a particular monitoring config without deleting it?”
Tell an agent
Update observability config {config_id} on {resource_type} {resource_id} to use rules {rule_ids}.
Set enabled to {enabled} on monitoring config {config_id} for agent {resource_id}, keeping rules {rule_ids}.
DELETE /evals/{resource_type}/{resource_id}/observability/configs/{config_id} Delete a specific observability config · DeleteObservabilityConfigV2 #
Ask an LLM
“How do I delete one monitoring config by its ID?”
“Can I remove a single observability config while keeping the others?”
Tell an agent destructive · confirm first
Delete observability config {config_id} from {resource_type} {resource_id}.
Remove monitoring config {config_id} on agent {resource_id}.
GET /evals/{resource_type}/{resource_id}/pre-publish/config Get the pre-publish eval config · GetPrePublishConfig #
Ask an LLM
“Which test sets must pass before my agent can be published?”
“Does my workforce have a pre-publish check configured?”
Tell an agent
Get the pre-publish config for {resource_type} {resource_id}.
Show the pre-publish test sets set on agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/pre-publish/config Set the pre-publish eval config · UpsertPrePublishConfig #
Ask an LLM
“How do I require certain test sets to run before an agent is published?”
“Does saving a pre-publish config replace the whole existing one?”
Tell an agent
Set the pre-publish config for {resource_type} {resource_id} to test sets {test_sets}.
Replace agent {resource_id}'s pre-publish checks with test sets {test_sets}.
DELETE /evals/{resource_type}/{resource_id}/pre-publish/config Remove the pre-publish eval config · DeletePrePublishConfig #
Ask an LLM
“Can I turn off pre-publish checks by deleting the config?”
“What removes the gate that runs test sets before publishing?”
Tell an agent destructive · confirm first
Delete the pre-publish config for {resource_type} {resource_id}.
Remove pre-publish checks from agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/pre-publish/trigger Run pre-publish checks on a draft version · TriggerPrePublishChecks #
Ask an LLM
“How do I test my agent's draft against its pre-publish test sets?”
“Does triggering pre-publish checks create one batch per test set?”
Tell an agent
Run pre-publish checks on draft {draft_version_id} of {resource_type} {resource_id}.
Trigger pre-publish evals for draft version {draft_version_id} of agent {resource_id}.
GET /evals/{resource_type}/{resource_id}/pre-publish/checks List pre-publish check attempts · ListPrePublishChecksV2 #
Ask an LLM
“Did my agent's last pre-publish check pass?”
“Can I see pre-publish results broken down by test set for a given version?”
Tell an agent
List pre-publish check attempts for {resource_type} {resource_id}.
Show pre-publish checks for version {version_id} of agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/observability/eval-scores/timeseries Chart observability eval scores over time · GetObservabilityEvalScoreTimeseriesForResource #
Ask an LLM
“How have my agent's live evaluation scores trended over the past month?”
“Can I chart eval scores for one observability config in my timezone?”
Tell an agent
Get the eval score timeseries for config {config_id} on {resource_type} {resource_id} from {start_date} to {end_date}.
Chart agent {resource_id}'s observability scores for config {config_id} since {start_date}.
GET /evals/{resource_type}/{resource_id}/test-cases/{test_case_id} Get a single test case · GetTestCaseForResource #
Ask an LLM
“What scenario and checks does a particular test case contain?”
“Can I look up one eval test case by ID?”
Tell an agent
Get test case {test_case_id} for {resource_type} {resource_id}.
Show the scenario of test case {test_case_id} on agent {resource_id}.
PUT /evals/{resource_type}/{resource_id}/test-cases/{test_case_id} Update a test case · UpdateTestCaseForResource #
Ask an LLM
“Can I edit the scenario or checks of an existing test case?”
“How would I rename a test case I already created?”
Tell an agent
Update test case {test_case_id} on {resource_type} {resource_id}: name {name}, scenario {scenario}, checks {check_ids}.
Change the scenario of existing test case {test_case_id} on agent {resource_id} to {scenario}.
DELETE /evals/{resource_type}/{resource_id}/test-cases/{test_case_id} Delete a test case · DeleteTestCaseForResource #
Ask an LLM
“How do I delete a test case I don't need anymore?”
“Can I remove one eval test case from an agent?”
Tell an agent destructive · confirm first
Delete test case {test_case_id} from {resource_type} {resource_id}.
Remove test case {test_case_id} on agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/test-cases/copy Copy a test case into another test set · CopyTestCaseForResource #
Ask an LLM
“Can I reuse a test case in a second test set without moving it?”
“What duplicates a test case into a different test set?”
Tell an agent
Copy test case {test_case_id} into test set {destination_test_set_id} on {resource_type} {resource_id}.
Duplicate test case {test_case_id} into set {destination_test_set_id} for agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/test-cases/move Move a test case to another test set · MoveTestCaseForResource #
Ask an LLM
“How do I move a test case out of one test set and into another?”
“Can I relocate a test case between sets on the same agent?”
Tell an agent
Move test case {test_case_id} from set {source_test_set_id} to set {destination_test_set_id} on {resource_type} {resource_id}.
Relocate test case {test_case_id} out of {source_test_set_id} into {destination_test_set_id} for agent {resource_id}.
POST /evals/{resource_type}/{resource_id}/test-sets Create a test set · CreateTestSetForResource #
Ask an LLM
“How do I group test cases into a new test set?”
“Can I create a test set and include existing test cases right away?”
Tell an agent
Create test set {name} for {resource_type} {resource_id}.
Create test set {name} on agent {resource_id} containing test cases {test_case_ids}.
GET /evals/{resource_type}/{resource_id}/test-sets List test sets for an agent or workforce · ListTestSetsForResource #
Ask an LLM
“What test sets exist for my agent?”
“Can I search an agent's test sets by name?”
Tell an agent
List test sets for {resource_type} {resource_id}.
Find test sets named like {search} on agent {resource_id}.
GET /evals/{resource_type}/{resource_id}/test-sets/{test_set_id} Get a single test set · GetTestSetForResource #
Ask an LLM
“What's inside a specific test set?”
“Can I retrieve one test set by its ID?”
Tell an agent
Get test set {test_set_id} for {resource_type} {resource_id}.
Show details of test set {test_set_id} on agent {resource_id}.
PUT /evals/{resource_type}/{resource_id}/test-sets/{test_set_id} Rename or reconfigure a test set · UpdateTestSetForResource #
Ask an LLM
“Can I rename an existing test set?”
“How do I change the tool simulation settings on a test set?”
Tell an agent
Rename test set {test_set_id} on {resource_type} {resource_id} to {name}.
Update test set {test_set_id} on agent {resource_id} with name {name} and simulation config {tool_simulation_config}.
DELETE /evals/{resource_type}/{resource_id}/test-sets/{test_set_id} Delete a test set · DeleteTestSetForResource #
Ask an LLM
“How do I delete a test set?”
“Can I delete a test set and its test cases together?”
Tell an agent destructive · confirm first
Delete test set {test_set_id} from {resource_type} {resource_id}.
Delete test set {test_set_id} on agent {resource_id}, including its test cases: {delete_test_cases}.

Documentation

📖
Documentation
https://relevanceai.com/docs
📖
APIReference
https://api-f1db6c.stack.tryrelevance.com/latest/documentation
📖
Authentication
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/authentication/relevance-ai-authentication.yml

Specifications

⚙
OpenAPI
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/openapi/relevance-ai-evals-api-openapi.yml

Other Resources

🔗
ErrorCatalog
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/errors/relevance-ai-problem-types.yml
🔗
DataModel
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/data-model/relevance-ai-data-model.yml

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/relevance-ai:relevance-ai-evals-api"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

relevance-ai-evals-api-openapi.yml Raw ↑ 
openapi: 3.2.0
info:
  description: Relevance AI Endpoints
  title: Relevance AI Endpoints Evals API
  version: latest
  contact:
    email: dev@tryrelevance.com
servers:
- url: /latest
security:
- AuthorizationHeader:
  - Authorization
tags:


# --- truncated at 32 KB (2124 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/openapi/relevance-ai-evals-api-openapi.yml