Relevance AI Evals API
The evals API from Relevance AI — 22 operation(s) for evals.
Operations 37
Each operation below carries the questions people ask an LLM about it and the instructions they give an agent to run it. Generated by API Evangelist overlay
POST
/evals/{resource_type}/{resource_id}/test-cases
Create a test case with a scenario and checks · CreateTestCase
#
Ask an LLM
“How do I write a new eval test case describing a scenario for my agent?”
“Can a new test case be added straight into a specific test set?”
Tell an agent
Create test case {name} for {resource_type} {resource_id} with scenario {scenario} and checks {check_ids}.
Add a test case named {name} to test set {test_set_id} on agent {resource_id}.
GET
/evals/{resource_type}/{resource_id}/test-cases
List test cases for an agent or workforce · ListTestCases
#
Ask an LLM
“What test cases have I written for my agent?”
“Can I list only the test cases in one test set?”
Tell an agent
List test cases for {resource_type} {resource_id}.
List test cases in test set {test_set_id} for {resource_type} {resource_id}.
GET
/evals/{resource_type}/{resource_id}/batches
List eval batches for an agent or workforce · ListEvalBatchByResourceNew
#
Ask an LLM
“Which evaluation batches have been run against my agent?”
“Can I sort an agent's eval batches by a field and direction?”
Tell an agent
List eval batches for {resource_type} {resource_id}.
Search eval batches on {resource_type} {resource_id} for {search}.
POST
/evals/{resource_type}/{resource_id}/rules
Create an evaluator rule · CreateEvalRuleForResource
#
Ask an LLM
“How do I add a new evaluator rule that grades my agent's conversations?”
“Can I define my own evaluation criteria for a workforce?”
Tell an agent
Create an evaluator rule on {resource_type} {resource_id} defined as {user_definition}.
Add a new eval rule to agent {resource_id}: {user_definition}.
GET
/evals/{resource_type}/{resource_id}/rules
List evaluator rules for an agent or workforce · ListEvalRulesForResource
#
Ask an LLM
“What evaluator rules are defined for my workforce?”
“Which grading rules is my agent being evaluated against?”
Tell an agent
List the evaluator rules for {resource_type} {resource_id}.
Show all eval rules configured on agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/runs
List eval runs for an agent or workforce · ListEvalRunsForResource
#
Ask an LLM
“What individual eval runs have been done on my agent recently?”
“Can I filter eval runs by a date range?”
Tell an agent
List eval runs for {resource_type} {resource_id}.
Show eval runs on {resource_type} {resource_id} within {date_range}.
POST
/evals/{resource_type}/{resource_id}/evaluate
Run an evaluation on an agent or workforce · EvaluateResource
#
Ask an LLM
“How do I evaluate a set of my agent's past conversations?”
“Can I run an evaluation using a whole test set and selected rules?”
Tell an agent
Start an evaluation run named {evaluation_run_name} on {resource_type} {resource_id} for conversations {conversation_ids}.
Evaluate agent {resource_id} against test set {test_set_id} as run {evaluation_run_name}.
GET
/evals/{resource_type}/{resource_id}/batches/{eval_batch_id}/summary
Get the summary of an eval batch · GetEvalBatchSummaryForResource
#
Ask an LLM
“How did my agent score in a particular eval batch?”
“Can I get summary results for one evaluation batch?”
Tell an agent
Summarize eval batch {eval_batch_id} for {resource_type} {resource_id}.
Show results summary for batch {eval_batch_id} on agent {resource_id}.
DELETE
/evals/{resource_type}/{resource_id}/batches/{eval_batch_id}
Delete an eval batch · DeleteEvalBatchForResource
#
Ask an LLM
“Can I delete an old evaluation batch I no longer need?”
“What removes an eval batch from an agent's history?”
Tell an agent destructive · confirm first
Delete eval batch {eval_batch_id} from {resource_type} {resource_id}.
Remove batch {eval_batch_id} from agent {resource_id}'s evals.
POST
/evals/{resource_type}/{resource_id}/batches/{eval_batch_id}/cancel
Cancel all pending runs in an eval batch · CancelEvalBatchForResource
#
Ask an LLM
“How do I stop an entire eval batch that's still running?”
“Does cancelling a batch stop both pending and running eval runs?”
Tell an agent destructive · confirm first
Cancel eval batch {eval_batch_id} on {resource_type} {resource_id}.
Stop every pending run in batch {eval_batch_id} for agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/runs/{eval_run_id}/cancel
Cancel a single eval run · CancelEvalRunForResource
#
Ask an LLM
“Can I cancel just one eval run without stopping the rest of the batch?”
“What stops a single stuck evaluation run?”
Tell an agent destructive · confirm first
Cancel eval run {eval_run_id} on {resource_type} {resource_id}.
Stop only run {eval_run_id} for agent {resource_id}.
PATCH
/evals/{resource_type}/{resource_id}/rules/{eval_rule_id}
Update an evaluator rule · UpdateEvalRuleForResource
#
Ask an LLM
“Can I change the definition of an existing evaluator rule?”
“How would I tweak the criteria of a rule already grading my agent?”
Tell an agent
Update evaluator rule {eval_rule_id} on {resource_type} {resource_id} to {user_definition}.
Change the definition of eval rule {eval_rule_id} for agent {resource_id}.
DELETE
/evals/{resource_type}/{resource_id}/rules/{eval_rule_id}
Delete an evaluator rule · DeleteEvalRuleForResource
#
Ask an LLM
“How do I remove an evaluator rule from my agent?”
“Can I delete a grading rule that's no longer relevant?”
Tell an agent destructive · confirm first
Delete evaluator rule {eval_rule_id} from {resource_type} {resource_id}.
Remove eval rule {eval_rule_id} on agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/observability/config
Create the legacy observability config · CreateObservabilityConfig
#
Ask an LLM
“How was the single observability config created before multiple configs were supported?”
“Can I still create an observability config via the deprecated endpoint?”
Tell an agent
Create the legacy single observability config on {resource_type} {resource_id} with rules {rule_ids}.
Use the deprecated endpoint to set up observability on agent {resource_id} using rules {rule_ids}.
PUT
/evals/{resource_type}/{resource_id}/observability/config
Update the legacy observability config · UpdateObservabilityConfig
#
Ask an LLM
“Can I still update an agent's observability settings through the older single-config endpoint?”
“What does the deprecated observability config update change?”
Tell an agent
Update the legacy observability config on {resource_type} {resource_id} to use rules {rule_ids}.
Set the deprecated observability config sample rate to {sample_rate} for agent {resource_id} with rules {rule_ids}.
GET
/evals/{resource_type}/{resource_id}/observability/config
Get the legacy observability config · GetObservabilityConfig
#
Ask an LLM
“Where is the older single observability config for an agent read from?”
“Is the deprecated get-observability-config endpoint still usable?”
Tell an agent
Get the legacy single observability config for {resource_type} {resource_id}.
Read the deprecated observability config on agent {resource_id}.
DELETE
/evals/{resource_type}/{resource_id}/observability/config
Delete the legacy observability config · DeleteObservabilityConfig
#
Ask an LLM
“Can I remove an agent's single observability config through the old endpoint?”
“What deletes the deprecated one-per-resource observability config?”
Tell an agent destructive · confirm first
Delete the legacy observability config on {resource_type} {resource_id}.
Remove the deprecated observability config from agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/observability/configs
Add an observability config · CreateObservabilityConfigV2
#
Ask an LLM
“How do I add another observability config so live conversations get evaluated?”
“Can I sample only a percentage of conversations for ongoing evaluation?”
Tell an agent
Add an observability config named {name} to {resource_type} {resource_id} with rules {rule_ids}.
Create a new monitoring config on agent {resource_id} sampling {sample_rate} with rules {rule_ids}.
GET
/evals/{resource_type}/{resource_id}/observability/configs
List observability configs for an agent or workforce · ListObservabilityConfigs
#
Ask an LLM
“Which observability configurations are set up on my workforce?”
“How many monitoring configs does my agent currently have?”
Tell an agent
List all observability configs for {resource_type} {resource_id}.
Show every monitoring config on agent {resource_id}.
PUT
/evals/{resource_type}/{resource_id}/observability/configs/{config_id}
Update a specific observability config · UpdateObservabilityConfigV2
#
Ask an LLM
“Can I change the rules or sample rate on one of several observability configs?”
“How do I disable a particular monitoring config without deleting it?”
Tell an agent
Update observability config {config_id} on {resource_type} {resource_id} to use rules {rule_ids}.
Set enabled to {enabled} on monitoring config {config_id} for agent {resource_id}, keeping rules {rule_ids}.
DELETE
/evals/{resource_type}/{resource_id}/observability/configs/{config_id}
Delete a specific observability config · DeleteObservabilityConfigV2
#
Ask an LLM
“How do I delete one monitoring config by its ID?”
“Can I remove a single observability config while keeping the others?”
Tell an agent destructive · confirm first
Delete observability config {config_id} from {resource_type} {resource_id}.
Remove monitoring config {config_id} on agent {resource_id}.
GET
/evals/{resource_type}/{resource_id}/pre-publish/config
Get the pre-publish eval config · GetPrePublishConfig
#
Ask an LLM
“Which test sets must pass before my agent can be published?”
“Does my workforce have a pre-publish check configured?”
Tell an agent
Get the pre-publish config for {resource_type} {resource_id}.
Show the pre-publish test sets set on agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/pre-publish/config
Set the pre-publish eval config · UpsertPrePublishConfig
#
Ask an LLM
“How do I require certain test sets to run before an agent is published?”
“Does saving a pre-publish config replace the whole existing one?”
Tell an agent
Set the pre-publish config for {resource_type} {resource_id} to test sets {test_sets}.
Replace agent {resource_id}'s pre-publish checks with test sets {test_sets}.
DELETE
/evals/{resource_type}/{resource_id}/pre-publish/config
Remove the pre-publish eval config · DeletePrePublishConfig
#
Ask an LLM
“Can I turn off pre-publish checks by deleting the config?”
“What removes the gate that runs test sets before publishing?”
Tell an agent destructive · confirm first
Delete the pre-publish config for {resource_type} {resource_id}.
Remove pre-publish checks from agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/pre-publish/trigger
Run pre-publish checks on a draft version · TriggerPrePublishChecks
#
Ask an LLM
“How do I test my agent's draft against its pre-publish test sets?”
“Does triggering pre-publish checks create one batch per test set?”
Tell an agent
Run pre-publish checks on draft {draft_version_id} of {resource_type} {resource_id}.
Trigger pre-publish evals for draft version {draft_version_id} of agent {resource_id}.
GET
/evals/{resource_type}/{resource_id}/pre-publish/checks
List pre-publish check attempts · ListPrePublishChecksV2
#
Ask an LLM
“Did my agent's last pre-publish check pass?”
“Can I see pre-publish results broken down by test set for a given version?”
Tell an agent
List pre-publish check attempts for {resource_type} {resource_id}.
Show pre-publish checks for version {version_id} of agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/observability/eval-scores/timeseries
Chart observability eval scores over time · GetObservabilityEvalScoreTimeseriesForResource
#
Ask an LLM
“How have my agent's live evaluation scores trended over the past month?”
“Can I chart eval scores for one observability config in my timezone?”
Tell an agent
Get the eval score timeseries for config {config_id} on {resource_type} {resource_id} from {start_date} to {end_date}.
Chart agent {resource_id}'s observability scores for config {config_id} since {start_date}.
GET
/evals/{resource_type}/{resource_id}/test-cases/{test_case_id}
Get a single test case · GetTestCaseForResource
#
Ask an LLM
“What scenario and checks does a particular test case contain?”
“Can I look up one eval test case by ID?”
Tell an agent
Get test case {test_case_id} for {resource_type} {resource_id}.
Show the scenario of test case {test_case_id} on agent {resource_id}.
PUT
/evals/{resource_type}/{resource_id}/test-cases/{test_case_id}
Update a test case · UpdateTestCaseForResource
#
Ask an LLM
“Can I edit the scenario or checks of an existing test case?”
“How would I rename a test case I already created?”
Tell an agent
Update test case {test_case_id} on {resource_type} {resource_id}: name {name}, scenario {scenario}, checks {check_ids}.
Change the scenario of existing test case {test_case_id} on agent {resource_id} to {scenario}.
DELETE
/evals/{resource_type}/{resource_id}/test-cases/{test_case_id}
Delete a test case · DeleteTestCaseForResource
#
Ask an LLM
“How do I delete a test case I don't need anymore?”
“Can I remove one eval test case from an agent?”
Tell an agent destructive · confirm first
Delete test case {test_case_id} from {resource_type} {resource_id}.
Remove test case {test_case_id} on agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/test-cases/copy
Copy a test case into another test set · CopyTestCaseForResource
#
Ask an LLM
“Can I reuse a test case in a second test set without moving it?”
“What duplicates a test case into a different test set?”
Tell an agent
Copy test case {test_case_id} into test set {destination_test_set_id} on {resource_type} {resource_id}.
Duplicate test case {test_case_id} into set {destination_test_set_id} for agent {resource_id}.
POST
/evals/{resource_type}/{resource_id}/test-cases/move
Move a test case to another test set · MoveTestCaseForResource
#
Ask an LLM
“How do I move a test case out of one test set and into another?”
“Can I relocate a test case between sets on the same agent?”
Tell an agent
Move test case {test_case_id} from set {source_test_set_id} to set {destination_test_set_id} on {resource_type} {resource_id}.
Relocate test case {test_case_id} out of {source_test_set_id} into {destination_test_set_id} for agent {resource_id}.
Ask an LLM
“How do I group test cases into a new test set?”
“Can I create a test set and include existing test cases right away?”
Tell an agent
Create test set {name} for {resource_type} {resource_id}.
Create test set {name} on agent {resource_id} containing test cases {test_case_ids}.
GET
/evals/{resource_type}/{resource_id}/test-sets
List test sets for an agent or workforce · ListTestSetsForResource
#
Ask an LLM
“What test sets exist for my agent?”
“Can I search an agent's test sets by name?”
Tell an agent
List test sets for {resource_type} {resource_id}.
Find test sets named like {search} on agent {resource_id}.
GET
/evals/{resource_type}/{resource_id}/test-sets/{test_set_id}
Get a single test set · GetTestSetForResource
#
Ask an LLM
“What's inside a specific test set?”
“Can I retrieve one test set by its ID?”
Tell an agent
Get test set {test_set_id} for {resource_type} {resource_id}.
Show details of test set {test_set_id} on agent {resource_id}.
PUT
/evals/{resource_type}/{resource_id}/test-sets/{test_set_id}
Rename or reconfigure a test set · UpdateTestSetForResource
#
Ask an LLM
“Can I rename an existing test set?”
“How do I change the tool simulation settings on a test set?”
Tell an agent
Rename test set {test_set_id} on {resource_type} {resource_id} to {name}.
Update test set {test_set_id} on agent {resource_id} with name {name} and simulation config {tool_simulation_config}.
DELETE
/evals/{resource_type}/{resource_id}/test-sets/{test_set_id}
Delete a test set · DeleteTestSetForResource
#
Ask an LLM
“How do I delete a test set?”
“Can I delete a test set and its test cases together?”
Tell an agent destructive · confirm first
Delete test set {test_set_id} from {resource_type} {resource_id}.
Delete test set {test_set_id} on agent {resource_id}, including its test cases: {delete_test_cases}.