Agenta

Agenta Evaluators API

Configure evaluators used to score variants.

Documentation GitHub OpenAPI

Operations 3

POST /evaluators/query List and filter evaluators. #

POST /evaluators/ Create an evaluator. #

GET /evaluators/catalog/templates/ List the built-in evaluator catalog templates. #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/agenta-evaluators-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: Agenta Applications Evaluators API
  description: Agenta is an open-source LLMOps platform for prompt management, LLM evaluation, and LLM observability. This specification documents the public cloud REST API surface used to manage applications and variants, fetch and deploy versioned prompt configurations, run evaluations and configure evaluators, manage testsets, and ingest and query observability traces. All endpoints are authenticated with an Agenta API key passed in the Authorization header. Agenta is MIT licensed and may also be self-hosted.
  termsOfService: https://agenta.ai/terms
  contact:
    name: Agenta Support
    url: https://agenta.ai/
    email: team@agenta.ai
  license:
    name: MIT
    url: https://github.com/Agenta-AI/agenta/blob/main/LICENSE
  version: '1.0'
servers:
- url: https://cloud.agenta.ai/api
  description: Agenta Cloud (US)
- url: https://eu.cloud.agenta.ai/api
  description: Agenta Cloud (EU)
security:
- ApiKeyAuth: []
tags:
- name: Evaluators
  description: Configure evaluators used to score variants.
paths:
  /evaluators/query:
    post:
      operationId: queryEvaluators
      tags:
      - Evaluators
      summary: List and filter evaluators.
      requestBody:
        required: false
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/EvaluatorQueryRequest'
      responses:
        '200':
          description: A list of evaluators.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Evaluator'
        '401':
          $ref: '#/components/responses/Unauthorized'
  /evaluators/:
    post:
      operationId: createEvaluator
      tags:
      - Evaluators
      summary: Create an evaluator.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/EvaluatorCreateRequest'
      responses:
        '200':
          description: The created evaluator.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Evaluator'
        '401':
          $ref: '#/components/responses/Unauthorized'
  /evaluators/catalog/templates/:
    get:
      operationId: listEvaluatorCatalogTemplates
      tags:
      - Evaluators
      summary: List the built-in evaluator catalog templates.
      responses:
        '200':
          description: A list of evaluator templates.
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/EvaluatorTemplate'
        '401':
          $ref: '#/components/responses/Unauthorized'
components:
  schemas:
    Error:
      type: object
      properties:
        detail:
          oneOf:
          - type: string
          - type: object
    Windowing:
      type: object
      description: Cursor / time-window pagination controls.
      properties:
        next:
          type: string
        limit:
          type: integer
          default: 100
        oldest:
          type: string
          format: date-time
        newest:
          type: string
          format: date-time
        order:
          type: string
          enum:
          - ascending
          - descending
    EvaluatorCreateRequest:
      type: object
      required:
      - evaluator
      properties:
        evaluator:
          type: object
          required:
          - slug
          properties:
            slug:
              type: string
            name:
              type: string
            data:
              type: object
              additionalProperties: true
    Evaluator:
      type: object
      properties:
        id:
          type: string
          format: uuid
        slug:
          type: string
        name:
          type: string
        data:
          type: object
          additionalProperties: true
        lifecycle:
          $ref: '#/components/schemas/Lifecycle'
    EvaluatorTemplate:
      type: object
      properties:
        key:
          type: string
          example: auto_exact_match
        name:
          type: string
        description:
          type: string
        settings_schema:
          type: object
          additionalProperties: true
    EvaluatorQueryRequest:
      type: object
      properties:
        include_archived:
          type: boolean
          default: false
        windowing:
          $ref: '#/components/schemas/Windowing'
    Lifecycle:
      type: object
      description: Audit metadata attached to most Agenta resources.
      properties:
        created_at:
          type: string
          format: date-time
        updated_at:
          type: string
          format: date-time
        created_by_id:
          type: string
          format: uuid
        updated_by_id:
          type: string
          format: uuid
  responses:
    Unauthorized:
      description: Missing or invalid API key.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  securitySchemes:
    ApiKeyAuth:
      type: http
      scheme: bearer
      description: 'Agenta API key sent in the Authorization header. Generate keys from the Agenta web app under Settings > API keys. The value is passed as `Authorization: ApiKey <key>` (Bearer-style header credential).'