forecall-dev

forecall-dev Evaluations API

The Evaluations API from forecall-dev — 1 operation(s) for evaluations.

Documentation GitHub OpenAPI

Operations 1

GET /v1/evaluations/{id} How far an evaluation has got, and its metrics #

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this

find_apisBrowse and filter every API in the catalog.
get_api_artifactsOne API's artifacts, grouped by type.
get_openapiThe primary OpenAPI for this API.
find_similar_apisAPIs that look like this one.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This API

curl "https://apis.io/api/v1/apis/forecall-dev:forecall-dev-evaluations-api"

All apis

curl "https://apis.io/api/v1/apis?limit=25"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

OpenAPI Specification

openapi: 3.2.0
info:
  title: Forecall Evaluations API
  version: v1
  description: 'Save and read the lints of the MCP servers registered in your Forecall dashboard, and evaluate how models choose their tools. Every endpoint but this document needs an API key from the dashboard (API keys): `Authorization: Bearer fc_…`.'
servers:
- url: https://api.forecall.dev
security:
- apiKey: []
tags:
- name: Evaluations
paths:
  /v1/evaluations/{id}:
    get:
      operationId: getEvaluation
      summary: How far an evaluation has got, and its metrics
      description: Takes no units. Each model's metrics appear when its run is done.
      parameters:
      - name: id
        in: path
        required: true
        description: The evaluation's id, as its start answered.
        schema:
          type: string
          format: uuid
      responses:
        '200':
          description: The evaluation
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Evaluation'
        '401':
          description: '`unauthorized`'
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: '`not_found`'
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '429':
          description: '`rate_limited`'
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
      tags:
      - Evaluations
components:
  schemas:
    Error:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
              enum:
              - unauthorized
              - quota_exceeded
              - payment_failed
              - plan_has_no_evaluations
              - not_found
              - input_too_large
              - invalid_json
              - unrecognized_shape
              - invalid_tool
              - too_many_tools
              - snake_case_keys
              - duplicate_names
              - invalid_label
              - invalid_request
              - unknown_models
              - unknown_distractors
              - lang_not_in_plan
              - rate_limited
              - internal
            message:
              type: string
            detail:
              type: object
              description: 'Where the input went wrong: path, keys, names, count and limit…'
          required:
          - code
          - message
      required:
      - error
    EvaluationSettings:
      type: object
      properties:
        casesPerTool:
          type: integer
          description: 'Positive cases per tool: 8 on Pro, 20 on Team'
        lang:
          type: string
          enum:
          - en
          - ja
        distractors:
          type: array
          items:
            type: string
      required:
      - casesPerTool
      - lang
      - distractors
    Evaluation:
      type: object
      properties:
        id:
          type: string
          format: uuid
        name:
          type: string
        status:
          type: string
          enum:
          - queued
          - running
          - done
          - failed
        versionId:
          type: string
          format: uuid
        createdAt:
          type: string
          format: date-time
        settings:
          $ref: '#/components/schemas/EvaluationSettings'
        cases:
          type: integer
          description: 0 until the cases are written
        runs:
          type: array
          items:
            type: object
            properties:
              id:
                type: string
                format: uuid
              model:
                type: string
              status:
                type: string
                enum:
                - queued
                - running
                - done
                - failed
              units:
                type: integer
              answered:
                type: integer
                description: Cases answered so far
              startedAt:
                type:
                - string
                - 'null'
                format: date-time
              finishedAt:
                type:
                - string
                - 'null'
                format: date-time
              metrics:
                oneOf:
                - type: object
                  properties:
                    selectionAcc:
                      type:
                      - number
                      - 'null'
                    argValidity:
                      type:
                      - number
                      - 'null'
                    falseCallRate:
                      type:
                      - number
                      - 'null'
                    lookupRate:
                      type:
                      - number
                      - 'null'
                      description: 'Of the cases written for a tool, the share that called a lookup tool (one that reads, such as list, get or search) instead: the model went to get an id or a snapshot first'
                    confusion:
                      type: object
                      description: Expected tool → the tool called → cases
                    counts:
                      type: object
                  required:
                  - selectionAcc
                  - argValidity
                  - falseCallRate
                  - lookupRate
                  - confusion
                  - counts
                - type: 'null'
                description: Null until the run is done; a rate is null when nothing was divided
            required:
            - id
            - model
            - status
            - units
            - answered
            - startedAt
            - finishedAt
            - metrics
      required:
      - id
      - name
      - status
      - versionId
      - createdAt
      - settings
      - cases
      - runs
  securitySchemes:
    apiKey:
      type: http
      scheme: bearer
      description: 'An API key: fc_ci_… or fc_vendor_…'