openapi: 3.0.1
info:
title: langfuse AnnotationQueues UnstableEvaluationRules API
version: ''
description: '## Authentication
Authenticate with the API using [Basic Auth](https://en.wikipedia.org/wiki/Basic_access_authentication), get API keys in the project settings:
- username: Langfuse Public Key
- password: Langfuse Secret Key
## Exports
- OpenAPI spec: https://cloud.langfuse.com/generated/api/openapi.yml'
tags:
- name: UnstableEvaluationRules
paths:
/api/public/unstable/evaluation-rules:
post:
description: 'Create an evaluation rule.
An evaluation rule defines **what** incoming data should be evaluated and **how prompt variables should be populated** from that data.
Use this resource after choosing an evaluator from the evaluator endpoints.
Key rules:
- `name` must be unique within the project for public evaluation rules
- `target` must be `observation` or `experiment`
- `evaluator.name` + `evaluator.scope` must identify an existing evaluator family returned by the evaluator endpoints
- Langfuse resolves that family to its latest version before saving the evaluation rule
- for `target=experiment`, use dataset `id` values from `GET /api/public/v2/datasets` when filtering by `datasetId`
- every evaluator prompt variable must be mapped exactly once
- `expected_output` mappings are only valid for `target=experiment`
- if `enabled=true`, Langfuse validates that the referenced evaluator can currently run
- at most 50 evaluation rules can be effectively active in one project at the same time
If an evaluation rule with the same `name` already exists in the project, the API returns `409`.
In that case, update the existing resource with `PATCH /api/public/unstable/evaluation-rules/{evaluationRuleId}` instead of creating a second one.
If enabling this resource would exceed the 50-active limit, the API also returns `409`.
In that case, disable or pause another active evaluation rule before enabling a new one.
Current scope:
- evaluation rules are live-ingestion rules only
- they do not trigger historical backfills
Recovery guidance:
- `400 invalid_filter_value`: fix the filter `column` or `value` using `details.column`, `details.invalidValues`, and `details.allowedValues`
- `400 invalid_filter_value` with `details.column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response
- `400 missing_variable_mapping`: fetch the evaluator again and make sure every variable in `variables` appears exactly once in `mapping`
- `400 duplicate_variable_mapping`: remove repeated mappings for the same variable
- `400 invalid_variable_mapping`: switch to a valid `source` for the selected `target`, or fix the variable name
- `400 invalid_json_path`: remove or correct the `jsonPath`
- `422 evaluator_preflight_failed`: the selected evaluator cannot run with the resolved model configuration. Fix the evaluator/default model setup, then retry the create request.'
operationId: unstable_evaluationRules_create
tags:
- UnstableEvaluationRules
parameters: []
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRule'
examples:
CreateObservationEvaluationRule:
value:
id: erule_123
name: answer-correctness-live
evaluator:
id: evaltmpl_123
name: answer-correctness
scope: project
target: observation
enabled: true
status: active
sampling: 1
filter:
- type: stringOptions
column: type
operator: any of
value:
- GENERATION
mapping:
- variable: input
source: input
- variable: output
source: output
createdAt: '2026-03-30T09:20:00.000Z'
updatedAt: '2026-03-30T09:20:00.000Z'
CreateExperimentEvaluationRule:
value:
id: erule_456
name: experiment-expected-output-match
evaluator:
id: evaltmpl_456
name: expected-output-match
scope: project
target: experiment
enabled: true
status: active
sampling: 0.5
filter:
- type: stringOptions
column: datasetId
operator: any of
value:
- 550e8400-e29b-41d4-a716-446655440000
mapping:
- variable: output
source: output
- variable: expected_output
source: expected_output
createdAt: '2026-03-30T09:30:00.000Z'
updatedAt: '2026-03-30T09:30:00.000Z'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'409':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'422':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/unstableCreateEvaluationRuleRequest'
examples:
CreateObservationEvaluationRule:
value:
name: answer-correctness-live
evaluator:
name: answer-correctness
scope: project
target: observation
enabled: true
sampling: 1
filter:
- type: stringOptions
column: type
operator: any of
value:
- GENERATION
mapping:
- variable: input
source: input
- variable: output
source: output
CreateExperimentEvaluationRule:
value:
name: experiment-expected-output-match
evaluator:
name: expected-output-match
scope: project
target: experiment
enabled: true
sampling: 0.5
filter:
- type: stringOptions
column: datasetId
operator: any of
value:
- 550e8400-e29b-41d4-a716-446655440000
mapping:
- variable: output
source: output
- variable: expected_output
source: expected_output
get:
description: 'List evaluation rules in the authenticated project.
Each item describes one live evaluation rule and its effective runtime status.'
operationId: unstable_evaluationRules_list
tags:
- UnstableEvaluationRules
parameters:
- name: page
in: query
description: 1-based page number. Defaults to `1`.
required: false
schema:
type: integer
nullable: true
- name: limit
in: query
description: Maximum number of items per page. Defaults to `50`.
required: false
schema:
type: integer
nullable: true
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRules'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
/api/public/unstable/evaluation-rules/{evaluationRuleId}:
get:
description: 'Get one evaluation rule by its identifier.
Use this endpoint to inspect the current evaluator, target, mapping, filters, and effective runtime status.'
operationId: unstable_evaluationRules_get
tags:
- UnstableEvaluationRules
parameters:
- name: evaluationRuleId
in: path
description: Evaluation rule identifier returned by the evaluation rule endpoints.
required: true
schema:
type: string
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRule'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
patch:
description: 'Update an evaluation rule.
Typical uses:
- enable or disable live execution
- switch to another evaluator
- adjust sampling
- change filters
- update variable mappings
Important behavior:
- provide only the fields you want to change
- if you provide `evaluator`, Langfuse resolves that evaluator family to its latest version before saving
- changing `target`, `filter`, or `mapping` must still produce a valid target-specific configuration
- if you change `target`, also send a compatible `filter` and `mapping` in the same request unless the existing ones are still valid for the new target
- if the resulting config is enabled, Langfuse re-validates that the selected evaluator can run
- if the update would move a non-active evaluation rule into the active state and the project already has 50 active evaluation rules, the API returns `409`
Recovery guidance:
- if the update fails with `missing_variable_mapping` or `invalid_variable_mapping` after changing `evaluator` or `target`, resend the request with a complete new `mapping`
- if the update fails with `invalid_filter_value` after changing `target`, resend the request with a target-compatible `filter`'
operationId: unstable_evaluationRules_update
tags:
- UnstableEvaluationRules
parameters:
- name: evaluationRuleId
in: path
description: Evaluation rule identifier.
required: true
schema:
type: string
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableEvaluationRule'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'422':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/unstableUpdateEvaluationRuleRequest'
delete:
description: 'Delete an evaluation rule.
This removes the live-ingestion rule only. It does not delete the referenced evaluator.'
operationId: unstable_evaluationRules_delete
tags:
- UnstableEvaluationRules
parameters:
- name: evaluationRuleId
in: path
description: Evaluation rule identifier.
required: true
schema:
type: string
responses:
'200':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstableDeleteEvaluationRuleResponse'
'400':
description: ''
content:
application/json:
schema: {}
'401':
description: ''
content:
application/json:
schema: {}
'403':
description: ''
content:
application/json:
schema: {}
'404':
description: ''
content:
application/json:
schema: {}
'405':
description: ''
content:
application/json:
schema: {}
'429':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/unstablePublicApiError'
security:
- BasicAuth: []
components:
schemas:
unstableEvaluationRuleTarget:
title: unstableEvaluationRuleTarget
type: string
enum:
- observation
- experiment
description: "The ingestion object type that should trigger evaluation runs.\n\nChoose the target first, because it changes both the valid filter columns and the valid variable-mapping sources:\n- `observation` evaluates live-ingested observations such as generations, spans, and events.\n It supports mapping from `input`, `output`, and `metadata`.\n- `experiment` evaluates live experiment executions and can additionally map `expected_output`.\n It currently supports filtering by `datasetId`.\n Discover valid dataset IDs with `GET /api/public/v2/datasets`, then use the returned dataset `id` values in your filter."
utilsMetaResponse:
title: utilsMetaResponse
type: object
properties:
page:
type: integer
description: current page number
limit:
type: integer
description: number of items per page
totalItems:
type: integer
description: number of total items given the current filters/selection (if any)
totalPages:
type: integer
description: number of total pages given the current limit
required:
- page
- limit
- totalItems
- totalPages
unstableEvaluationRules:
title: unstableEvaluationRules
type: object
description: Paginated list of evaluation rules.
properties:
data:
type: array
items:
$ref: '#/components/schemas/unstableEvaluationRule'
description: Evaluation rules in the current page.
meta:
$ref: '#/components/schemas/utilsMetaResponse'
description: Standard pagination metadata.
required:
- data
- meta
unstableDeleteEvaluationRuleResponse:
title: unstableDeleteEvaluationRuleResponse
type: object
description: Confirmation response returned after successful deletion.
properties:
message:
type: string
description: Always `Evaluation rule successfully deleted`.
required:
- message
unstableEvaluationRuleFilter:
title: unstableEvaluationRuleFilter
oneOf:
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- datetime
- $ref: '#/components/schemas/unstableDateTimeEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- string
- $ref: '#/components/schemas/unstableStringEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- number
- $ref: '#/components/schemas/unstableNumberEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- stringOptions
- $ref: '#/components/schemas/unstableStringOptionsEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- categoryOptions
- $ref: '#/components/schemas/unstableCategoryOptionsEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- arrayOptions
- $ref: '#/components/schemas/unstableArrayOptionsEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- stringObject
- $ref: '#/components/schemas/unstableStringObjectEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- numberObject
- $ref: '#/components/schemas/unstableNumberObjectEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- boolean
- $ref: '#/components/schemas/unstableBooleanEvaluationRuleFilter'
required:
- type
- type: object
allOf:
- type: object
properties:
type:
type: string
enum:
- 'null'
- $ref: '#/components/schemas/unstableNullEvaluationRuleFilter'
required:
- type
description: "One filter condition used to decide whether a live-ingested target should be evaluated.\n\nAn evaluation rule can include zero or more filter objects. All filters must be satisfied for the target to run.\n\nHow to build a valid filter object:\n- Pick the `target` first, because it changes the supported columns.\n- Pick the filter `type`. That determines which fields are required.\n- Use `key` only for object filters such as `metadata`.\n- Use the correct `value` shape for the chosen filter `type`.\n\nOperator quick reference by filter `type`:\n- `string`: `\"=\"`, `contains`, `does not contain`, `starts with`, `ends with`\n- `number`: `\"=\"`, `\">\"`, `\"<\"`, `\">=\"`, `\"<=\"`\n- `datetime`: `\"=\"`, `\">\"`, `\"<\"`, `\">=\"`, `\"<=\"`\n- `stringOptions`: `any of`, `none of`\n- `arrayOptions`: `any of`, `none of`, `all of`\n- `stringObject`: same operators as `string`\n- `null`: `is null`, `is not null`\n\nSupported columns by target:\n- `target=observation`\n - `type`: `stringOptions`, operators `any of` / `none of`, values `GENERATION`, `SPAN`, `EVENT`\n - `name`: `stringOptions`, operators `any of` / `none of`\n - `environment`: `stringOptions`, operators `any of` / `none of`\n - `level`: `stringOptions`, operators `any of` / `none of`, values `DEBUG`, `DEFAULT`, `WARNING`, `ERROR`\n - `version`: `string`\n - `traceName`: `stringOptions`, operators `any of` / `none of`\n - `userId`: `string`\n - `sessionId`: `string`\n - `tags`: `arrayOptions`, operators `any of` / `none of` / `all of`\n - `metadata`: `stringObject` with `key`\n - `parentObservationId`: `null`, operators `is null` / `is not null`\n - `calledToolNames`: `arrayOptions`, operators `any of` / `none of` / `all of`\n - `toolCalls`: `number`\n- `target=experiment`\n - `datasetId`: `stringOptions`, operators `any of` / `none of`\n Use dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.\n\nRecovery guidance:\n- `invalid_filter_value` with `details.column` but no `invalidValues`: the selected `column` is not supported for the chosen `target`\n- `invalid_filter_value` with `details.invalidValues`: the selected values are not allowed for that column. Replace them with one of `details.allowedValues` when provided.\n- `invalid_filter_value` for `column=datasetId`: call `GET /api/public/v2/datasets`, then retry with dataset `id` values from that response."
unstableEvaluationRuleOptionsFilterOperator:
title: unstableEvaluationRuleOptionsFilterOperator
type: string
enum:
- any of
- none of
unstableDateTimeEvaluationRuleFilter:
title: unstableDateTimeEvaluationRuleFilter
type: object
properties:
column:
type: string
description: Column to filter on.
operator:
$ref: '#/components/schemas/unstableEvaluationRuleNumberFilterOperator'
description: Comparison operator for datetime values.
value:
type: string
format: date-time
description: Datetime value to compare against.
required:
- column
- operator
- value
unstableCreateEvaluationRuleRequest:
title: unstableCreateEvaluationRuleRequest
type: object
description: 'Request body for creating an evaluation rule.
Checklist for agents and SDK clients:
- reference an existing evaluator family by `evaluator.name` and `evaluator.scope`
- choose `target=observation` or `target=experiment`
- if `target=experiment` and you want a dataset filter, call `GET /api/public/v2/datasets` first and use dataset `id` values in `filter[].value`
- fetch or inspect the evaluator first, then provide a complete variable mapping for every evaluator variable listed in `variables`
- optionally narrow execution with `filter`
- set `enabled=true` only when you want live execution immediately'
properties:
name:
type: string
description: Human-readable deployment name.
example: answer-correctness-live
evaluator:
$ref: '#/components/schemas/unstableEvaluationRuleEvaluatorReference'
description: 'Evaluator family to use.
Use `name` and `scope` from the evaluator endpoints.
Langfuse resolves that family to its latest version before saving the rule.'
target:
$ref: '#/components/schemas/unstableEvaluationRuleTarget'
description: Target object type to evaluate.
enabled:
type: boolean
description: Whether the deployment should be active immediately after creation.
example: true
sampling:
type: number
format: double
nullable: true
description: Optional sampling fraction. Defaults to `1`.
filter:
type: array
items:
$ref: '#/components/schemas/unstableEvaluationRuleFilter'
nullable: true
description: 'Optional filter list.
Omit or pass an empty list to evaluate all matching targets for the selected `target`.
Each filter object must use a column that is valid for that `target`.
For `target=experiment`, `column=datasetId` expects dataset `id` values from `GET /api/public/v2/datasets`, not dataset names.'
mapping:
type: array
items:
$ref: '#/components/schemas/unstableEvaluationRuleMapping'
description: 'Required variable mappings.
Every evaluator variable must appear exactly once.
Build this list from the evaluator `variables` array returned by the evaluator endpoints.'
required:
- name
- evaluator
- target
- enabled
- mapping
unstableEvaluationRule:
title: unstableEvaluationRule
type: object
description: 'Live evaluation rule for incoming data.
An evaluation rule answers:
- which evaluator should be used
- which target objects should trigger scoring
- how often scoring should run
- which target fields should populate each evaluator variable
- whether the deployment is active, inactive, or paused
Important status semantics:
- `enabled` is the desired on/off setting from the client
- `status` is the effective runtime state after Langfuse applies validation and blocking rules
- `enabled=true` with `status=paused` means the rule should run, but Langfuse has paused it until the underlying problem is fixed'
properties:
id:
type: string
description: Stable evaluation rule identifier.
example: erule_123
name:
type: string
description: Human-readable deployment name. This is independent from the evaluator name.
example: answer-correctness-live
evaluator:
$ref: '#/components/schemas/unstableEvaluationRuleEvaluator'
description: 'Evaluator currently used by this rule.
`name` and `scope` identify the evaluator family conceptually.
`id` is the currently active evaluator version in that family.
If you create a newer project version with the same evaluator name later, existing evaluation rules are moved to it automatically.'
target:
$ref: '#/components/schemas/unstableEvaluationRuleTarget'
description: Target object type that should trigger scoring.
enabled:
type: boolean
description: Desired enabled state configured by the client.
example: true
status:
$ref: '#/components/schemas/unstableEvaluationRuleStatus'
description: Effective runtime status after Langfuse applies validation and blocking rules.
pausedReason:
type: string
nullable: true
description: Machine-readable reason when `status=paused`, otherwise `null`.
pausedMessage:
type: string
nullable: true
description: Human-readable explanation when `status=paused`, otherwise `null`.
sampling:
type: number
format: double
description: 'Fraction of matching target objects that should be evaluated.
Must be greater than `0` and less than or equal to `1`.
- `1` means evaluate every matching target.
- `0.25` means evaluate approximately 25% of matching targets.'
example: 1
filter:
type: array
items:
$ref: '#/components/schemas/unstableEvaluationRuleFilter'
description: List of filter conditions used to decide whether a target should be evaluated.
mapping:
type: array
items:
$ref: '#/components/schemas/unstableEvaluationRuleMapping'
description: Variable mappings used to populate the evaluator prompt from the live target object.
createdAt:
type: string
format: date-time
description: Timestamp when the evaluation rule was created.
example: '2026-03-30T09:20:00.000Z'
updatedAt:
type: string
format: date-time
description: Timestamp when the evaluation rule was last updated.
example: '2026-03-30T09:20:00.000Z'
required:
- id
- name
- evaluator
- target
- enabled
- status
- sampling
- filter
- mapping
- createdAt
- updatedAt
unstableEvaluationRuleStringFilterOperator:
title: unstableEvaluationRuleStringFilterOperator
type: string
enum:
- '='
- contains
- does not contain
- starts with
- ends with
unstableEvaluationRuleMappingSource:
title: unstableEvaluationRuleMappingSource
type: string
enum:
- input
- output
- metadata
- expected_output
description: 'Source field used to populate a prompt variable.
Use these values when mapping evaluator prompt variables to live data.
Target-specific rules:
- `target=observation` supports `input`, `output`, and `metadata`
- `target=experiment` supports `input`, `output`, `metadata`, and `expected_output`
Source semantics:
- `input`: the observation or experiment input payload
- `output`: the observation or experiment output payload
- `metadata`: the metadata object for the target. Combine with `jsonPath` when you need one nested field instead of the whole object.
- `expected_output`: the experiment item''s expected output. Only vali
# --- truncated at 32 KB (49 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/langfuse/refs/heads/main/openapi/langfuse-unstableevaluationrules-api-openapi.yml