OpenAI

OpenAI Evals API

Manage and run evals in the OpenAI platform.

Documentation GitHub OpenAPI

Operations 12

Each operation below carries the questions people ask an LLM about it and the instructions they give an agent to run it. Generated by API Evangelist overlay

GET /evals List evaluations in a project · List evals #

Ask an LLM

“Which evals have I set up in this project?”

“Can I sort my evaluations by when they were last updated?”

Tell an agent

List the evaluations in my project.

Show my evals ordered by {order_by}.

POST /evals Create an evaluation · Create eval #

Ask an LLM

“How do I define an eval to test how well a model performs on my data?”

“What do I need to specify when setting up a new evaluation's grading criteria?”

Tell an agent

Create an eval named {name} with data source config {data_source_config} and testing criteria {testing_criteria}.

Set up a new evaluation using graders {testing_criteria} over data shaped like {data_source_config}.

GET /evals/{eval_id} Retrieve an evaluation · Get an eval #

Ask an LLM

“What testing criteria does one of my evals use?”

“Can I look up the configuration of an eval by its ID?”

Tell an agent

Get eval {eval_id}.

Show the testing criteria and data source config of eval {eval_id}.

POST /evals/{eval_id} Rename or retag an evaluation · Update an eval #

Ask an LLM

“Can I rename an evaluation after I've created it?”

“Which properties of an existing eval am I allowed to change?”

Tell an agent

Rename eval {eval_id} to {name}.

Update the metadata on eval {eval_id} to {metadata}.

DELETE /evals/{eval_id} Delete an evaluation · Delete an eval #

Ask an LLM

“How do I remove an evaluation I no longer need?”

“Can I delete an entire eval definition by ID?”

Tell an agent destructive · confirm first

Delete eval {eval_id}.

Permanently remove the evaluation {eval_id} and its definition.

GET /evals/{eval_id}/runs List the runs of an evaluation · Get eval runs #

Ask an LLM

“What runs have been executed against one of my evals?”

“Can I see only the eval runs that failed or were canceled?”

Tell an agent

List the runs for eval {eval_id}.

Show runs of eval {eval_id} with status {status}.

POST /evals/{eval_id}/runs Start an evaluation run · Create eval run #

Ask an LLM

“How do I kick off a run to test a model against an existing eval?”

“Can I choose which data source and model config an eval run uses?”

Tell an agent

Start a run of eval {eval_id} using data source {data_source}.

Kick off an eval run named {name} on eval {eval_id} with data source {data_source}.

GET /evals/{eval_id}/runs/{run_id} Check an evaluation run · Get an eval run #

Ask an LLM

“Has my eval run finished, and what were the results?”

“Can I fetch one evaluation run to see its status and scores?”

Tell an agent

Get run {run_id} of eval {eval_id}.

Show the status and results of eval run {run_id} on {eval_id}.

POST /evals/{eval_id}/runs/{run_id} Cancel an ongoing evaluation run · Cancel eval run #

Ask an LLM

“Can I stop an eval run that's still going?”

“How do I cancel an evaluation run I started with the wrong config?”

Tell an agent destructive · confirm first

Cancel eval run {run_id} on eval {eval_id}.

Stop the ongoing run {run_id} of evaluation {eval_id}.

DELETE /evals/{eval_id}/runs/{run_id} Delete an evaluation run · Delete eval run #

Ask an LLM

“Can I delete a single run from an eval while keeping the eval itself?”

“What's the way to clean up old eval runs I don't need?”

Tell an agent destructive · confirm first

Delete eval run {run_id} from eval {eval_id}.

Remove the old run {run_id} but keep evaluation {eval_id}.

GET /evals/{eval_id}/runs/{run_id}/output_items List output items from an evaluation run · Get eval run output items #

Ask an LLM

“Which individual test cases passed or failed in an eval run?”

“Can I list just the failed output items from a run?”

Tell an agent

List the output items of run {run_id} on eval {eval_id}.

Show the {status} output items from eval run {run_id} of {eval_id}.

GET /evals/{eval_id}/runs/{run_id}/output_items/{output_item_id} Retrieve one output item from an evaluation run · Get an output item of an eval run #

Ask an LLM

“What did the model output and how was it graded for one specific test case?”

“Can I inspect a single eval output item by its ID?”

Tell an agent

Get output item {output_item_id} from run {run_id} of eval {eval_id}.

Show the grading details for output item {output_item_id} in eval run {run_id} on {eval_id}.

OpenAI Evals API

Operations 12

Documentation

Specifications

Schemas & Data

Other Resources

Work with this as data

MCP server

Tools for apis

Call it yourself

Get an API key

OpenAPI Specification