Every API here is available over the APIs.io API and to AI agents over MCP.
openapi: 3.2.0
info:
title: langfuse AnnotationQueues Unstable Evaluation Rules API
version: '1.0'
description: '## Authentication
Authenticate with the API using [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication), get API keys in the project settings:
- username: Langfuse Public Key
- password: Langfuse Secret Key
## Exports
- OpenAPI spec: https://cloud.langfuse.com/generated/api/openapi.yml'
servers:
- url: https://cloud.langfuse.com/api/public
description: Base URL declared by the provider in apis.yml (roadmap#122).
tags:
- name: UnstableEvaluationRules
paths:
/api/public/unstable/evaluation-rules:
post:
description: 'Create an evaluation rule.
An evaluation rule defines **what** incoming data should be evaluated and **how prompt variables should be populated** from that data.
Use this resource after choosing an evaluator from the evaluator endpoints.
Key rules:
- `name` must be unique within the project for public evaluation rules
- `target` must be `observation` or `experiment`
- `evaluator.name` + `evaluator.scope` must identify an existing evaluator family returned by the evaluator endpoints
- Langfuse resolves that family to its latest version before saving the evaluation rule
- for `target=experiment`, use dataset `id` values from `GET /api/public/v2/datasets` when filtering by `datasetId`
- every evaluator prompt variable must be mapped exactly once
- `expected_output` mappings are only valid for `target=experiment`
- if `enabled=true`, Langfuse validates that the referenced evaluator can currently run
- at most 50 evaluation rules can be effectively active in one project at the same time
If an evaluation rule with the same `name` already exists in the project, the API returns `409`.
In that case, update the existing resource with `PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of creating a second one.
If enabling this resource would exceed the 50-active limit, the API also returns `409`.
In that case, disable or pause another active evaluation rule before enabling a new one.
Current scope:
- evaluation rules are live-ingestion rules only
- they do not trigger historical backfills
Recovery guidance:
- `400 invalid_filter_value`: fix the filter `column` or `value` using `details.column`, `details.invalidValues`, and `details.allowedValues`
- `400 invalid_filter_value` with `details.column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response
- `400 missing_variable_mapping`: fetch the evaluator again and make sure every variable in `variables` appears exactly once in `mapping`
- `400 duplicate_variable_mapping`: remove repeated mappings for the same variable
- `400 invalid_variable_mapping`: switch to a valid `source` for the selected `target`, or fix the variable name
- `400 invalid_json_path`: remove or correct the `jsonPath`
- `422 evaluator_preflight_failed`: the selected evaluator cannot run with the resolved model configuration. Fix the evaluator/default model setup, then retry the create request.'
operationId: unstable_evaluationRules_create
tags:
- UnstableEvaluationRules
parameters: []
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRule'
examples:
CreateObservationEvaluationRule:
value:
id: erule_123
name: answer-correctness-live
evaluator:
id: evaltmpl_123
name: answer-correctness
scope: project
target: observation
enabled: true
status: active
sampling: 1
filter:
- type: stringOptions
column: type
operator: any of
value:
- GENERATION
mapping:
- variable: input
source: input
- variable: output
source: output
createdAt: '2026-03-30T09:20:00.000Z'
updatedAt: '2026-03-30T09:20:00.000Z'
CreateExperimentEvaluationRule:
value:
id: erule_456
name: experiment-expected-output-match
evaluator:
id: evaltmpl_456
name: expected-output-match
scope: project
target: experiment
enabled: true
status: active
sampling: 0.5
filter:
- type: stringOptions
column: datasetId
operator: any of
value:
- 550e8400-e29b-41d4-a716-446655440000
mapping:
- variable: output
source: output
- variable: expected_output
source: expected_output
createdAt: '2026-03-30T09:30:00.000Z'
updatedAt: '2026-03-30T09:30:00.000Z'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'409':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'422':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/unstableCreateEvaluationRuleRequest'
examples:
CreateObservationEvaluationRule:
value:
name: answer-correctness-live
evaluator:
name: answer-correctness
scope: project
target: observation
enabled: true
sampling: 1
filter:
- type: stringOptions
column: type
operator: any of
value:
- GENERATION
mapping:
- variable: input
source: input
- variable: output
source: output
CreateExperimentEvaluationRule:
value:
name: experiment-expected-output-match
evaluator:
name: expected-output-match
scope: project
target: experiment
enabled: true
sampling: 0.5
filter:
- type: stringOptions
column: datasetId
operator: any of
value:
- 550e8400-e29b-41d4-a716-446655440000
mapping:
- variable: output
source: output
- variable: expected_output
source: expected_output
get:
description: 'List evaluation rules in the authenticated project.
Each item describes one live evaluation rule and its effective runtime status.'
operationId: unstable_evaluationRules_list
tags:
- UnstableEvaluationRules
parameters:
- name: page
in: query
description: 1-based page number. Defaults to `1`.
required: false
schema:
type:
- integer
- 'null'
- name: limit
in: query
description: Maximum number of items per page. Defaults to `50`.
required: false
schema:
type:
- integer
- 'null'
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRules'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
/api/public/unstable/evaluation-rules/{evaluationRuleId}:
get:
description: 'Get one evaluation rule by its identifier.
Use this endpoint to inspect the current evaluator, target, mapping, filters, and effective runtime status.'
operationId: unstable_evaluationRules_get
tags:
- UnstableEvaluationRules
parameters:
- name: evaluationRuleId
in: path
description: Evaluation rule identifier returned by the evaluation rule endpoints.
required: true
schema:
type: string
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRule'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
patch:
description: 'Update an evaluation rule.
Typical uses:
- enable or disable live execution
- switch to another evaluator
- adjust sampling
- change filters
- update variable mappings
Important behavior:
- provide only the fields you want to change
- if you provide `evaluator`, Langfuse resolves that evaluator family to its latest version before saving
- changing `target`, `filter`, or `mapping` must still produce a valid target-specific configuration
- if you change `target`, also send a compatible `filter` and `mapping` in the same request unless the existing ones are still valid for the new target
- if the resulting config is enabled, Langfuse re-validates that the selected evaluator can run
- if the update would move a non-active evaluation rule into the active state and the project already has 50 active evaluation rules, the API returns `409`
Recovery guidance:
- if the update fails with `missing_variable_mapping` or `invalid_variable_mapping` after changing `evaluator` or `target`, resend the request with a complete new `mapping`
- if the update fails with `invalid_filter_value` after changing `target`, resend the request with a target-compatible `filter`'
operationId: unstable_evaluationRules_update
tags:
- UnstableEvaluationRules
parameters:
- name: evaluationRuleId
in: path
description: Evaluation rule identifier.
required: true
schema:
type: string
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRule'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'422':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/unstableUpdateEvaluationRuleRequest'
delete:
description: 'Delete an evaluation rule.
This removes the live-ingestion rule only. It does not delete the referenced evaluator.'
operationId: unstable_evaluationRules_delete
tags:
- UnstableEvaluationRules
parameters:
- name: evaluationRuleId
in: path
description: Evaluation rule identifier.
required: true
schema:
type: string
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableDeleteEvaluationRuleResponse'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
components:
schemas:
unstableEvaluationRuleFilter:
title: unstableEvaluationRuleFilter
oneOf:
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- datetime
- $ref: '#/components/schemas/unstableDateTimeEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- string
- $ref: '#/components/schemas/unstableStringEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- number
- $ref: '#/components/schemas/unstableNumberEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- stringOptions
- $ref: '#/components/schemas/unstableStringOptionsEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- categoryOptions
- $ref: '#/components/schemas/unstableCategoryOptionsEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- arrayOptions
- $ref: '#/components/schemas/unstableArrayOptionsEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- stringObject
- $ref: '#/components/schemas/unstableStringObjectEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- numberObject
- $ref: '#/components/schemas/unstableNumberObjectEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- boolean
- $ref: '#/components/schemas/unstableBooleanEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- 'null'
- $ref: '#/components/schemas/unstableNullEvaluationRuleFilter'
required:
- type
description: "One filter condition used to decide whether a live-ingested target should be evaluated.\n\nAn evaluation rule can include zero or more filter objects. All filters must be satisfied for the target to run.\n\nHow to build a valid filter object:\n- Pick the `target` first, because it changes the supported columns.\n- Pick the filter `type`. That determines which fields are required.\n- Use `key` only for object filters such as `metadata`.\n- Use the correct `value` shape for the chosen filter `type`.\n\nOperator quick reference by filter `type`:\n- `string`: `\"=\"`, `contains`, `does not contain`, `starts with`, `ends with`\n- `number`: `\"=\"`, `\">\"`, `\"<\"`, `\">=\"`, `\"<=\"`\n- `datetime`: `\"=\"`, `\">\"`, `\"<\"`, `\">=\"`, `\"<=\"`\n- `stringOptions`: `any of`, `none of`\n- `arrayOptions`: `any of`, `none of`, `all of`\n- `stringObject`: same operators as `string`\n- `null`: `is null`, `is not null`\n\nSupported columns by target:\n- `target=observation`\n - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`\n - `name`: `stringOptions`, operators `any of` / `none of`\n - `environment`: `stringOptions`, operators `any of` / `none of`\n - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`\n - `version`: `string`\n - `traceName`: `stringOptions`, operators `any of` / `none of`\n - `userId`: `string`\n - `sessionId`: `string`\n - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`\n - `metadata`: `stringObject` with `key`\n - `parentObservationId`: `null`, operators `is null` / `is not null`\n - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`\n - `toolCalls`: `number`\n- `target=experiment`\n - `datasetId`: `stringOptions`, operators `any of` / `none of`\n Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.\n\nRecovery guidance:\n- `invalid_filter_value` with `details.column` but no `invalidValues`: the selected `column` is not supported for the chosen `target`\n- `invalid_filter_value` with `details.invalidValues`: the selected values are not allowed for that column. Replace them with one of `details.allowedValues` when provided.\n- `invalid_filter_value` for `column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response."
unstableNumberEvaluationRuleFilter:
title: unstableNumberEvaluationRuleFilter
type: object
properties:
column:
type: string
description: Column to filter on.
operator:
$ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'
value:
type: number
format: double
required:
- column
- operator
- value
unstablePublicApiErrorCode:
title: unstablePublicApiErrorCode
type: string
enum:
- authentication_failed
- access_denied
- invalid_request
- invalid_query
- invalid_body
- invalid_filter_value
- invalid_json_path
- invalid_variable_mapping
- missing_variable_mapping
- duplicate_variable_mapping
- resource_not_found
- name_conflict
- evaluator_preflight_failed
- conflict
- unprocessable_content
- rate_limited
- method_not_allowed
- internal_error
description: 'Machine-readable error code returned by the unstable evaluators API.
SDKs, CLIs, and agents should branch on `code` rather than parsing the human-readable `message`.
The HTTP status still indicates the broad error class, while `code` gives the specific failure reason.'
unstableStringOptionsEvaluationRuleFilter:
title: unstableStringOptionsEvaluationRuleFilter
type: object
properties:
column:
type: string
description: Column to filter on.
operator:
$ref: '#/components/schemas/unstableEvaluationRuleOptionsFilterOperator'
value:
type: array
items:
type: string
description: One or more allowed string values.
required:
- column
- operator
- value
unstableBooleanEvaluationRuleFilter:
title: unstableBooleanEvaluationRuleFilter
type: object
properties:
column:
type: string
description: Column to filter on.
operator:
$ref: '#/components/schemas/unstableEvaluationRuleBooleanFilterOperator'
value:
type: boolean
required:
- column
- operator
- value
unstablePublicApiError:
title: unstablePublicApiError
type: object
description: 'Standard error envelope for the unstable evaluators API.
Response handling guidance:
- Use the HTTP status code for the broad class of failure.
- Use `code` for precise branching in SDKs, CLIs, or agents.
- Inspect `details` for field-level validation context such as invalid filter values, malformed JSONPath expressions, or missing variable mappings.
- Retry only after fixing the specific issue described by `code` and `details`.'
properties:
message:
type: string
description: Human-readable description of the failure.
example: 'Filter column "type" contains unsupported value(s): INVALID'
code:
$ref: '#/components/schemas/unstablePublicApiErrorCode'
description: Stable machine-readable error code.
details:
$ref: '#/components/schemas/unstablePublicApiErrorDetails'
description: Optional structured error context. Inspect the populated fields based on `code`.
required:
- message
- code
unstableEvaluationRuleStatus:
title: unstableEvaluationRuleStatus
type: string
enum:
- active
- inactive
- paused
description: 'Effective runtime status of the evaluation rule.
- `active`: enabled and currently runnable.
- `inactive`: disabled by configuration.
- `paused`: enabled, but Langfuse has blocked execution until the underlying issue is resolved.'
unstableEvaluationRuleEvaluatorReference:
title: unstableEvaluationRuleEvaluatorReference
type: object
description: 'Evaluator family reference used when creating or updating an evaluation rule.
`name` and `scope` are enough to identify the evaluator family in the authenticated project context.'
properties:
name:
type: string
description: Evaluator family name.
scope:
$ref: '#/components/schemas/unstableEvaluatorScope'
description: Whether the evaluator family is project-owned or Langfuse-managed.
required:
- name
- scope
unstablePublicApiErrorDetails:
title: unstablePublicApiErrorDetails
type: object
description: 'Optional structured context attached to an unstable-evals error.
The populated fields depend on the error `code`:
- request parsing failures populate `issues`
- filter validation failures populate `field`, `column`, `invalidValues`, and `allowedValues`
- variable mapping failures populate `field`, `variable`, or `variables`
- JSONPath validation failures populate `field`, `variable`, and `value`
- evaluator preflight failures populate `evaluatorName`, `provider`, and `model`
- rate limiting populates `retryAfterSeconds`, `limit`, `remaining`, and `resetAt`'
properties:
issues:
type:
- array
- 'null'
items:
$ref: '#/components/schemas/unstablePublicApiValidationIssue'
description: Validation issues for malformed request bodies or query parameters.
field:
type:
- string
- 'null'
description: Path-like reference to the failing field, for example `mapping[1].jsonPath`.
column:
type:
- string
- 'null'
description: Filter column that failed validation.
invalidValues:
type:
- array
- 'null'
items:
type: string
description: Unsupported values supplied by the caller.
allowedValues:
type:
- array
- 'null'
items:
type: string
description: Allowed values for the failing filter column.
variable:
type:
- string
- 'null'
description: Evaluator variable involved in the failure.
variables:
type:
- array
- 'null'
items:
type: string
description: Multiple evaluator variables involved in the failure, for example missing mappings.
value:
type:
- string
- 'null'
description: Raw invalid value supplied by the caller.
evaluatorName:
type:
- string
- 'null'
description: Evaluator name used during preflight validation.
provider:
type:
- string
- 'null'
description: Provider resolved during evaluator preflight, if any.
model:
type:
- string
- 'null'
description: Model resolved during evaluator preflight, if any.
retryAfterSeconds:
type:
- integer
- 'null'
description: Suggested retry delay for rate-limited requests.
limit:
type:
- integer
- 'null'
description: Numeric limit associated with the failure, for example the active evaluation-rule cap or the current rate-limit window.
remaining:
type:
- integer
- 'null'
description: Remaining requests in the current rate-limit window.
resetAt:
type:
- string
- 'null'
description: ISO-8601 timestamp when the current rate-limit window resets.
unstableEvaluationRule:
title: unstableEvaluationRule
type: object
description: 'Live evaluation rule for incoming data.
An evaluation rule answers:
- which evaluator should be used
- which target objects should trigger scoring
- how often scoring should run
- which target fields should populate each evaluator variable
- whether the deployment is active, inactive, or paused
Important status semantics:
- `enabled` is the desired on/off setting from the client
- `status` is the effective runtime state after Langfuse applies validation and blocking rules
- `enabled=true` with `status=paused` means the rule should run, but Langfuse has paused it until the underlying problem is fixed'
properties:
id:
type: string
description: Stable evaluation rule identifier.
example: erule_123
name:
type: string
description: Human-readable deployment name. This is independent from the evaluator name.
example: answer-correctness-live
evaluator:
$ref: '#/components/schemas/unstableEvaluationRuleEvaluator'
description: 'Evaluator currently used by this rule.
`name` and `scope` identify the evaluator family conceptually.
`id` is the currently active evaluator version in that family.
If you create a newer project version with the same evaluator name later, existing evaluation rules are moved to it automatically.'
target:
$ref: '#/components/schemas/unstableEvaluationRuleTarget'
description: Target object type that should trigger scoring.
enabled:
type: boolean
description: Desired enabled state configured by the client.
example: true
status:
$ref: '#/components/schemas/unstableEvaluationRuleStatus'
description: Effective runtime status after Langfuse applies validation and blocking rules.
pausedReason:
type:
- string
- 'null'
description: Machine-readable reason when `status=paused`, otherwise `null`.
pausedMessage:
type:
- string
- 'null'
description: Human-readable explanation when `status=paused`, otherwise `null`.
sampling:
type: number
format: double
description: 'Fraction of matching target objects that should be evaluated.
Must be greater than `0` and less than or equal to `1`.
# --- truncated at 32 KB (49 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/langfuse/refs/heads/main/openapi/langfuse-unstableevaluationrules-api-openapi.yml