Happyrobot

Happyrobot Custom Evals API

The Custom Evals API from Happyrobot — 8 operation(s) for custom evals.

GitHub OpenAPI

Operations 11

GET /nodes/{node_id}/custom-evals List custom evals for a prompt node
POST /nodes/{node_id}/custom-evals Create a custom eval for a prompt node
GET /nodes/{node_id}/custom-evals/tools Get available tools for custom evals
GET /nodes/{node_id}/custom-evals/default-variables Get default variables for custom evals
POST /nodes/{node_id}/custom-evals/extract-from-run Extract custom eval from a run
POST /nodes/{node_id}/custom-evals/folders Create a custom eval folder for a prompt node
GET /custom-evals/{eval_id} Get a custom eval by ID
PATCH /custom-evals/{eval_id} Update a custom eval
DELETE /custom-evals/{eval_id} Delete a custom eval
POST /custom-evals/{eval_id}/run Run a custom eval
GET /custom-evals/{eval_id}/runs List custom eval runs

Documentation

📖
Documentation
https://docs.happyrobot.ai/api-reference
📖
APIReference
https://docs.happyrobot.ai/api-reference
📖
Authentication
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/authentication/happyrobot-authentication.yml

Specifications

⚙
OpenAPI
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/openapi/happyrobot-custom-evals-api-openapi.yml

Other Resources

🔗
Conventions
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/conventions/happyrobot-conventions.yml
🔗
ErrorCatalog
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/errors/happyrobot-problem-types.yml
🔗
DataModel
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/data-model/happyrobot-data-model.yml
🔗
Webhooks
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/asyncapi/happyrobot-events.yml
🔗
AgenticAccess
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/agentic-access/happyrobot-agentic-access.yml
🔗
Lifecycle
https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/lifecycle/happyrobot-lifecycle.yml

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/happyrobot-custom-evals-api"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

happyrobot-custom-evals-api-openapi.yml Raw ↑ 
openapi: 3.2.0
info:
  title: Happyrobot Public Custom Evals API
  description: Public API endpoints for Happyrobot
  version: 0.1.1
servers:
- url: https://platform.happyrobot.ai/api/v2
security:
- bearerAuth: []
tags:
- name: Custom Evals
paths:
  /nodes/{node_id}/custom-evals:
    get:
      summary: List custom evals for a prompt node
      tags:
      - Custom Evals
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: node_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '200':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  data:
                    type: array
                    items:
                      type: object
                      properties:
                        id:
                          type: string
                        org_id:
                          type: string
                        workflow_id:
                          type:
                          - string
                          - 'null'
                        prompt_node_id:
                          type: string
                        prompt_node_persistent_id:
                          type:
                          - string
                          - 'null'
                        name:
                          type: string
                        description:
                          type:
                          - string
                          - 'null'
                        test_messages:
                          type: array
                          items:
                            type: object
                            additionalProperties: {}
                        expected_response:
                          type:
                          - string
                          - 'null'
                        judge_model:
                          type:
                          - string
                          - 'null'
                        expected_tool_calls:
                          type:
                          - array
                          - 'null'
                          items:
                            type: object
                            additionalProperties: {}
                        variables:
                          type: object
                          additionalProperties: {}
                        source_run_id:
                          type:
                          - string
                          - 'null'
                        is_deleted:
                          type: boolean
                        eval_mode:
                          type: string
                        examples:
                          type: array
                          items:
                            type: object
                            additionalProperties: {}
                        northstar_ids:
                          type: array
                          items:
                            type: string
                        folder_id:
                          type:
                          - string
                          - 'null'
                        created_at:
                          type: string
                        updated_at:
                          type: string
                      required:
                      - id
                      - org_id
                      - workflow_id
                      - prompt_node_id
                      - prompt_node_persistent_id
                      - name
                      - description
                      - test_messages
                      - expected_response
                      - judge_model
                      - expected_tool_calls
                      - variables
                      - source_run_id
                      - is_deleted
                      - eval_mode
                      - examples
                      - northstar_ids
                      - folder_id
                      - created_at
                      - updated_at
                      additionalProperties: false
                  folders:
                    type: array
                    items:
                      type: object
                      properties:
                        id:
                          type: string
                        name:
                          type: string
                      required:
                      - id
                      - name
                      additionalProperties: false
                required:
                - data
                - folders
                additionalProperties: false
        '401':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '404':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '500':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
    post:
      summary: Create a custom eval for a prompt node
      tags:
      - Custom Evals
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                name:
                  type: string
                  minLength: 1
                test_messages:
                  type: array
                  items:
                    type: object
                    properties:
                      role:
                        type: string
                        enum:
                        - user
                        - assistant
                        - tool
                      content:
                        type: string
                      turn_index:
                        type: integer
                        minimum: -9007199254740991
                        maximum: 9007199254740991
                      tool_calls:
                        type: array
                        items:
                          type: object
                          properties:
                            function:
                              type: object
                              properties:
                                name:
                                  type: string
                                arguments:
                                  type: string
                              required:
                              - name
                              - arguments
                            id:
                              type: string
                          required:
                          - function
                      tool_call_id:
                        type: string
                    required:
                    - role
                    - content
                    - turn_index
                description:
                  type: string
                expected_response:
                  type: string
                judge_model:
                  default: gpt-5.4-nano-medium
                  type: string
                expected_tool_calls:
                  type: array
                  items:
                    type: object
                    properties:
                      name:
                        type: string
                      args_description:
                        type: object
                        additionalProperties:
                          type: string
                    required:
                    - name
                variables:
                  type: object
                  additionalProperties: {}
                source_run_id:
                  type: string
                  format: uuid
                  pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
                eval_mode:
                  type: string
                  enum:
                  - custom
                  - northstar
                northstar_ids:
                  type: array
                  items:
                    type: string
                    format: uuid
                    pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
              required:
              - name
              - test_messages
        required: true
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: node_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '201':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  test:
                    type: object
                    properties:
                      id:
                        type: string
                      org_id:
                        type: string
                      workflow_id:
                        type:
                        - string
                        - 'null'
                      prompt_node_id:
                        type: string
                      prompt_node_persistent_id:
                        type:
                        - string
                        - 'null'
                      name:
                        type: string
                      description:
                        type:
                        - string
                        - 'null'
                      test_messages:
                        type: array
                        items:
                          type: object
                          additionalProperties: {}
                      expected_response:
                        type:
                        - string
                        - 'null'
                      judge_model:
                        type:
                        - string
                        - 'null'
                      expected_tool_calls:
                        type:
                        - array
                        - 'null'
                        items:
                          type: object
                          additionalProperties: {}
                      variables:
                        type: object
                        additionalProperties: {}
                      source_run_id:
                        type:
                        - string
                        - 'null'
                      is_deleted:
                        type: boolean
                      eval_mode:
                        type: string
                      examples:
                        type: array
                        items:
                          type: object
                          additionalProperties: {}
                      northstar_ids:
                        type: array
                        items:
                          type: string
                      folder_id:
                        type:
                        - string
                        - 'null'
                      created_at:
                        type: string
                      updated_at:
                        type: string
                    required:
                    - id
                    - org_id
                    - workflow_id
                    - prompt_node_id
                    - prompt_node_persistent_id
                    - name
                    - description
                    - test_messages
                    - expected_response
                    - judge_model
                    - expected_tool_calls
                    - variables
                    - source_run_id
                    - is_deleted
                    - eval_mode
                    - examples
                    - northstar_ids
                    - folder_id
                    - created_at
                    - updated_at
                    additionalProperties: false
                required:
                - test
                additionalProperties: false
        '400':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '401':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '404':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '500':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
  /nodes/{node_id}/custom-evals/tools:
    get:
      summary: Get available tools for custom evals
      tags:
      - Custom Evals
      description: Available tool/function definitions for this node — for building expected_tool_calls.
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: node_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '200':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  tools:
                    type: array
                    items:
                      type: object
                      properties:
                        id:
                          type: string
                        name:
                          type: string
                        function:
                          type: object
                          additionalProperties: {}
                      required:
                      - id
                      - name
                      - function
                      additionalProperties: false
                required:
                - tools
                additionalProperties: false
        '401':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '404':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '500':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
  /nodes/{node_id}/custom-evals/default-variables:
    get:
      summary: Get default variables for custom evals
      tags:
      - Custom Evals
      description: Merged default variable values available at runtime (org + use-case + workflow levels).
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: query
        name: version_id
        required: true
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: node_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '200':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  variables:
                    type: object
                    additionalProperties: {}
                required:
                - variables
                additionalProperties: false
        '401':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '404':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '500':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
  /nodes/{node_id}/custom-evals/extract-from-run:
    post:
      summary: Extract custom eval from a run
      tags:
      - Custom Evals
      description: Build a custom eval from an existing run. Auto-extracts messages, variables, and suggests a name.
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                run_id:
                  type: string
                  format: uuid
                  pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
                message_id:
                  type: string
                  format: uuid
                  pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
              required:
              - run_id
              - message_id
        required: true
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: node_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '200':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  messages:
                    type: array
                    items:
                      type: object
                      additionalProperties: {}
                  variables:
                    type: object
                    additionalProperties: {}
                  suggested_name:
                    type: string
                  source_run_id:
                    type: string
                  navigation_info:
                    type: object
                    properties:
                      prompt_node_id:
                        type: string
                      workflow_id:
                        type: string
                    required:
                    - prompt_node_id
                    - workflow_id
                    additionalProperties: false
                required:
                - messages
                - variables
                - suggested_name
                - source_run_id
                - navigation_info
                additionalProperties: false
        '401':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '404':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '500':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
  /nodes/{node_id}/custom-evals/folders:
    post:
      summary: Create a custom eval folder for a prompt node
      tags:
      - Custom Evals
      description: Create a folder for organizing this prompt node's custom evals. File evals into it via folder_id on update.
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                name:
                  type: string
                  minLength: 1
              required:
              - name
        required: true
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: node_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '201':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  folder:
                    type: object
                    properties:
                      id:
                        type: string
                      name:
                        type: string
                    required:
                    - id
                    - name
                    additionalProperties: false
                required:
                - folder
                additionalProperties: false
        '401':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '404':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
        '500':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  error:
                    type: string
                  message:
                    type: string
                  statusCode:
                    type: integer
                    minimum: -9007199254740991
                    maximum: 9007199254740991
                  details: {}
                required:
                - error
                additionalProperties: false
  /custom-evals/{eval_id}:
    get:
      summary: Get a custom eval by ID
      tags:
      - Custom Evals
      parameters:
      - schema:
          type: string
          format: uuid
          pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$
        in: path
        name: eval_id
        required: true
      security:
      - bearerAuth: []
      responses:
        '200':
          description: Default Response
          content:
            application/json:
              schema:
                type: object
                properties:
                  test:
                    type: object
                    properties:
                      id:
                        type: string
                      org_id:
                        type: string
                      workflow_id:
                        type:
                        - string
                        - 'null'
                      prompt_node_id:
                        type: string
                      prompt_node_persistent_id:
                        type:
                        - string
                        - 'null'
                      name:
                        type: string
                      description:
                        type:
                        - string
                        - 'null'
                      test_messages:
                        type: array
                        items:
                          type: object
                          additionalProperties: {}
                      expected_response:
                        type:
                        - string
                        - 'null'
                      judge_model:
                        type:
                        - string
                        - 'null'
                      expected_tool_calls:
                        type:
                        - array
                        - 'null'
                        items:
                          type: object
                          additionalProperties: {}
                      variables:
                        type: object
                        additionalProperties: {}
            

# --- truncated at 32 KB (55 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/happyrobot/refs/heads/main/openapi/happyrobot-custom-evals-api-openapi.yml