PromptLayer

PromptLayer Evaluations & Datasets API

Create, run, and score evaluation reports against datasets.

Documentation GitHub OpenAPI

Operations 6

GET /evaluations List evaluation reports #

POST /evaluations Create an evaluation report #

GET /evaluations/{id} Get an evaluation report #

POST /evaluations/{id}/run Run an evaluation report #

GET /evaluations/{id}/score Get an evaluation report score #

GET /datasets List datasets #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/promptlayer-evaluations-datasets-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: PromptLayer Evaluations & Datasets API
  description: 'REST API for PromptLayer, a prompt engineering, prompt management, and LLM observability platform. The API logs and tracks LLM requests, manages a versioned prompt registry with release labels, ingests OpenTelemetry-style spans and traces, and runs evaluations against datasets. All requests are authenticated with an `X-API-KEY` header. Note: some legacy tracking endpoints are served under the `/rest` path prefix while newer endpoints are served at the API root.'
  termsOfService: https://www.promptlayer.com/terms-of-service
  contact:
    name: PromptLayer Support
    email: hello@promptlayer.com
  version: '1.0'
servers:
- url: https://api.promptlayer.com
security:
- ApiKeyAuth: []
tags:
- name: Evaluations & Datasets
  description: Create, run, and score evaluation reports against datasets.
paths:
  /evaluations:
    get:
      operationId: listEvaluations
      tags:
      - Evaluations & Datasets
      summary: List evaluation reports
      description: Lists evaluation reports in the workspace.
      parameters:
      - name: workspace_id
        in: query
        required: false
        schema:
          type: integer
      - name: per_page
        in: query
        required: false
        schema:
          type: integer
      - name: page
        in: query
        required: false
        schema:
          type: integer
      responses:
        '200':
          description: A list of evaluation reports
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/EvaluationList'
        '401':
          description: Unauthorized
    post:
      operationId: createEvaluation
      tags:
      - Evaluations & Datasets
      summary: Create an evaluation report
      description: Creates a new evaluation report (pipeline) over a dataset.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateEvaluationRequest'
      responses:
        '201':
          description: Evaluation report created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Evaluation'
        '401':
          description: Unauthorized
        '422':
          description: Validation error
  /evaluations/{id}:
    get:
      operationId: getEvaluation
      tags:
      - Evaluations & Datasets
      summary: Get an evaluation report
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: integer
      responses:
        '200':
          description: An evaluation report
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Evaluation'
        '401':
          description: Unauthorized
        '404':
          description: Not found
  /evaluations/{id}/run:
    post:
      operationId: runEvaluation
      tags:
      - Evaluations & Datasets
      summary: Run an evaluation report
      description: Executes an evaluation report, scoring its rows against the configured columns.
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: integer
      responses:
        '200':
          description: Evaluation run started
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SuccessMessage'
        '401':
          description: Unauthorized
        '404':
          description: Not found
  /evaluations/{id}/score:
    get:
      operationId: getEvaluationScore
      tags:
      - Evaluations & Datasets
      summary: Get an evaluation report score
      parameters:
      - name: id
        in: path
        required: true
        schema:
          type: integer
      responses:
        '200':
          description: Evaluation score
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/EvaluationScore'
        '401':
          description: Unauthorized
        '404':
          description: Not found
  /datasets:
    get:
      operationId: listDatasets
      tags:
      - Evaluations & Datasets
      summary: List datasets
      description: Lists evaluation datasets in the workspace.
      parameters:
      - name: workspace_id
        in: query
        required: false
        schema:
          type: integer
      responses:
        '200':
          description: A list of datasets
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DatasetList'
        '401':
          description: Unauthorized
components:
  schemas:
    Evaluation:
      type: object
      properties:
        id:
          type: integer
        name:
          type: string
        status:
          type: string
        dataset_id:
          type: integer
        workspace_id:
          type: integer
        created_at:
          type: string
          format: date-time
    SuccessMessage:
      type: object
      properties:
        success:
          type: boolean
        message:
          type: string
    EvaluationScore:
      type: object
      properties:
        id:
          type: integer
        score:
          type: number
        passed:
          type: integer
        failed:
          type: integer
    CreateEvaluationRequest:
      type: object
      required:
      - name
      properties:
        name:
          type: string
        dataset_id:
          type: integer
        workspace_id:
          type: integer
    DatasetList:
      type: object
      properties:
        items:
          type: array
          items:
            $ref: '#/components/schemas/Dataset'
        total:
          type: integer
    Dataset:
      type: object
      properties:
        id:
          type: integer
        name:
          type: string
        workspace_id:
          type: integer
        row_count:
          type: integer
    EvaluationList:
      type: object
      properties:
        items:
          type: array
          items:
            $ref: '#/components/schemas/Evaluation'
        total:
          type: integer
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-KEY