Keywords AI

Keywords AI Evaluators API

Output-scoring evaluators.

Documentation GitHub OpenAPI

Operations 3

POST /evaluators/ Create an evaluator #

POST /evaluators/list/ List evaluators #

POST /evaluators/{evaluator_id}/run/ Run an evaluator #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/keywordsai-evaluators-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: Keywords AI Datasets Evaluators API
  description: 'Keywords AI is an LLM observability and gateway platform. This specification documents the publicly documented REST surface served from https://api.keywordsai.co/api: the OpenAI-compatible chat completions proxy, the asynchronous request-logging endpoint, prompt and prompt-version management, threads, traces, evaluators, users (customers), datasets, and experiments. All endpoints authenticate with a Bearer API key. (Keywords AI is rebranding to Respan; the keywordsai.co host and API remain active.)'
  termsOfService: https://www.keywordsai.co/terms-of-service
  contact:
    name: Keywords AI Support
    url: https://www.keywordsai.co
    email: team@keywordsai.co
  version: '1.0'
servers:
- url: https://api.keywordsai.co/api
  description: Keywords AI production API base.
security:
- bearerAuth: []
tags:
- name: Evaluators
  description: Output-scoring evaluators.
paths:
  /evaluators/:
    post:
      operationId: createEvaluator
      tags:
      - Evaluators
      summary: Create an evaluator
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Evaluator'
      responses:
        '201':
          description: Evaluator created.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Evaluator'
  /evaluators/list/:
    post:
      operationId: listEvaluators
      tags:
      - Evaluators
      summary: List evaluators
      description: Retrieve evaluators matching the supplied filters.
      requestBody:
        required: false
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ListFilter'
      responses:
        '200':
          description: A paginated list of evaluators.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/PaginatedList'
  /evaluators/{evaluator_id}/run/:
    parameters:
    - name: evaluator_id
      in: path
      required: true
      schema:
        type: string
    post:
      operationId: runEvaluator
      tags:
      - Evaluators
      summary: Run an evaluator
      description: Execute an evaluator against supplied inputs/logs and return scores.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                input:
                  type: string
                output:
                  type: string
                metadata:
                  type: object
                  additionalProperties: true
      responses:
        '200':
          description: Evaluation result with score(s).
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/EvaluationResult'
components:
  schemas:
    ListFilter:
      type: object
      properties:
        filters:
          type: object
          additionalProperties: true
        page:
          type: integer
          default: 1
        page_size:
          type: integer
          default: 50
        sort_by:
          type: string
    Evaluator:
      type: object
      required:
      - name
      properties:
        id:
          type: string
        name:
          type: string
        type:
          type: string
          description: Evaluator type (e.g. llm, code, human).
        description:
          type: string
        definition:
          type: object
          additionalProperties: true
    EvaluationResult:
      type: object
      properties:
        evaluator_id:
          type: string
        score:
          type: number
        passed:
          type: boolean
        reason:
          type: string
    PaginatedList:
      type: object
      properties:
        count:
          type: integer
        next:
          type:
          - string
          - 'null'
        previous:
          type:
          - string
          - 'null'
        results:
          type: array
          items:
            type: object
            additionalProperties: true
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: 'Keywords AI API key supplied as the Authorization Bearer header (Authorization: Bearer <KEYWORDSAI_API_KEY>).'