Relevance AI

Relevance AI Evals API

The evals API from Relevance AI — 22 operation(s) for evals.

GitHub OpenAPI

Operations 37

POST /evals/{resource_type}/{resource_id}/test-cases CreateTestCase #
GET /evals/{resource_type}/{resource_id}/test-cases ListTestCases #
GET /evals/{resource_type}/{resource_id}/batches ListEvalBatchByResourceNew #
POST /evals/{resource_type}/{resource_id}/rules CreateEvalRuleForResource #
GET /evals/{resource_type}/{resource_id}/rules ListEvalRulesForResource #
POST /evals/{resource_type}/{resource_id}/runs ListEvalRunsForResource #
POST /evals/{resource_type}/{resource_id}/evaluate EvaluateResource #
GET /evals/{resource_type}/{resource_id}/batches/{eval_batch_id}/summary GetEvalBatchSummaryForResource #
DELETE /evals/{resource_type}/{resource_id}/batches/{eval_batch_id} DeleteEvalBatchForResource #
POST /evals/{resource_type}/{resource_id}/batches/{eval_batch_id}/cancel CancelEvalBatchForResource #
POST /evals/{resource_type}/{resource_id}/runs/{eval_run_id}/cancel CancelEvalRunForResource #
PATCH /evals/{resource_type}/{resource_id}/rules/{eval_rule_id} UpdateEvalRuleForResource #
DELETE /evals/{resource_type}/{resource_id}/rules/{eval_rule_id} DeleteEvalRuleForResource #
POST /evals/{resource_type}/{resource_id}/observability/config CreateObservabilityConfig #
PUT /evals/{resource_type}/{resource_id}/observability/config UpdateObservabilityConfig #
GET /evals/{resource_type}/{resource_id}/observability/config GetObservabilityConfig #
DELETE /evals/{resource_type}/{resource_id}/observability/config DeleteObservabilityConfig #
POST /evals/{resource_type}/{resource_id}/observability/configs CreateObservabilityConfigV2 #
GET /evals/{resource_type}/{resource_id}/observability/configs ListObservabilityConfigs #
PUT /evals/{resource_type}/{resource_id}/observability/configs/{config_id} UpdateObservabilityConfigV2 #
DELETE /evals/{resource_type}/{resource_id}/observability/configs/{config_id} DeleteObservabilityConfigV2 #
GET /evals/{resource_type}/{resource_id}/pre-publish/config GetPrePublishConfig #
POST /evals/{resource_type}/{resource_id}/pre-publish/config UpsertPrePublishConfig #
DELETE /evals/{resource_type}/{resource_id}/pre-publish/config DeletePrePublishConfig #
POST /evals/{resource_type}/{resource_id}/pre-publish/trigger TriggerPrePublishChecks #
GET /evals/{resource_type}/{resource_id}/pre-publish/checks ListPrePublishChecksV2 #
POST /evals/{resource_type}/{resource_id}/observability/eval-scores/timeseries GetObservabilityEvalScoreTimeseriesForResource #
GET /evals/{resource_type}/{resource_id}/test-cases/{test_case_id} GetTestCaseForResource #
PUT /evals/{resource_type}/{resource_id}/test-cases/{test_case_id} UpdateTestCaseForResource #
DELETE /evals/{resource_type}/{resource_id}/test-cases/{test_case_id} DeleteTestCaseForResource #
POST /evals/{resource_type}/{resource_id}/test-cases/copy CopyTestCaseForResource #
POST /evals/{resource_type}/{resource_id}/test-cases/move MoveTestCaseForResource #
POST /evals/{resource_type}/{resource_id}/test-sets CreateTestSetForResource #
GET /evals/{resource_type}/{resource_id}/test-sets ListTestSetsForResource #
GET /evals/{resource_type}/{resource_id}/test-sets/{test_set_id} GetTestSetForResource #
PUT /evals/{resource_type}/{resource_id}/test-sets/{test_set_id} UpdateTestSetForResource #
DELETE /evals/{resource_type}/{resource_id}/test-sets/{test_set_id} DeleteTestSetForResource #

Documentation

📖
Documentation
https://relevanceai.com/docs
📖
APIReference
https://api-f1db6c.stack.tryrelevance.com/latest/documentation
📖
Authentication
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/authentication/relevance-ai-authentication.yml

Specifications

⚙
OpenAPI
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/openapi/relevance-ai-evals-api-openapi.yml

Other Resources

🔗
ErrorCatalog
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/errors/relevance-ai-problem-types.yml
🔗
DataModel
https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/data-model/relevance-ai-data-model.yml

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/relevance-ai-evals-api"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

relevance-ai-evals-api-openapi.yml Raw ↑ 
openapi: 3.2.0
info:
  description: Relevance AI Endpoints
  title: Relevance AI Endpoints Evals API
  version: latest
  contact:
    email: dev@tryrelevance.com
servers:
- url: /latest
security:
- AuthorizationHeader:
  - Authorization
tags:


# --- truncated at 32 KB (2123 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/relevance-ai/refs/heads/main/openapi/relevance-ai-evals-api-openapi.yml