Bem Function Accuracy API
Monitor, evaluate, and iterate on the quality of every function in your environment. Function Accuracy bundles two complementary loops: ## Evaluations (`/v3/eval`) Trigger and retrieve per-transformation evaluations. Evaluations run asynchronously and score each transformation's output against the function's schema for confidence, per-field hallucination detection, and relevance. Supported for `extract`, `transform`, `analyze`, and `join` events. 1. **Trigger** — `POST /v3/eval` queues jobs for a batch of transformation IDs. 2. **Poll** — `GET /v3/eval/results` returns the current state of each requested ID, partitioned into `results`, `pending`, and `failed`. Accepts either `eventIDs` (preferred) or `transformationIDs` as a comma-separated query parameter, and always keys the response by event KSUID. Up to 100 IDs may be submitted per request. ## Metrics, review, regression (`/v3/functions/{metrics,review,regression,compare}`) Roll evaluation results and user corrections up into actionable function-level signal: - **`GET /v3/functions/metrics`** — aggregate accuracy, precision, recall, F1, and confusion-matrix counts per function. - **`POST /v3/functions/review`** — sample-size estimation, confidence-bucketed distribution, PR-AUC, and per-threshold confidence intervals (Wald or Wilson) for picking review cutoffs. - **`POST /v3/functions/regression`** — replay corrected historical inputs against a new function version, producing a labeled regression dataset. - **`POST /v3/functions/regression/corrections`** — propagate baseline corrections onto the regression dataset so it can be scored. - **`POST /v3/functions/compare`** — compute aggregate and field-level lift between any two versions, optionally scoped to the regression dataset. All five endpoints support `extract` end-to-end on both the vision and OCR paths, alongside the legacy `transform` / `analyze` / `join` types.
openapi: 3.1.0
info:
title: Bem Buckets Function Accuracy API
version: 1.0.0
description: "Buckets are named partitions of the knowledge graph within an\naccount+environment. Entities, mentions, and relations are scoped to a\nbucket so a single account+environment can host multiple isolated graphs\n— for example one per data source or workspace.\n\nEvery account+environment has exactly one **default** bucket, used by\nunscoped flows. The default bucket can be renamed but never deleted.\n\nUse these endpoints to create, list, fetch, rename, and delete buckets:\n\n- **`POST /v3/buckets`** creates a non-default bucket.\n- **`GET /v3/buckets`** lists buckets with cursor pagination\n (`startingAfter` / `endingBefore` over `bucketID`).\n- **`PATCH /v3/buckets/{bucketID}`** updates `name` and/or `description`.\n- **`DELETE /v3/buckets/{bucketID}`** soft-deletes a bucket. A non-empty\n bucket is rejected with `409 Conflict` unless `?cascade=true` is\n passed; the default bucket can never be deleted."
servers:
- url: https://api.bem.ai
description: US Region API
variables: {}
- url: https://api.eu1.bem.ai
description: EU Region API
variables: {}
security:
- API Key: []
tags:
- name: Function Accuracy
description: "Monitor, evaluate, and iterate on the quality of every function in your\nenvironment. Function Accuracy bundles two complementary loops:\n\n## Evaluations (`/v3/eval`)\n\nTrigger and retrieve per-transformation evaluations. Evaluations run\nasynchronously and score each transformation's output against the\nfunction's schema for confidence, per-field hallucination detection,\nand relevance. Supported for `extract`, `transform`, `analyze`, and\n`join` events.\n\n1. **Trigger** — `POST /v3/eval` queues jobs for a batch of transformation IDs.\n2. **Poll** — `GET /v3/eval/results` returns the current state of each\n requested ID, partitioned into `results`, `pending`, and `failed`.\n Accepts either `eventIDs` (preferred) or `transformationIDs` as a\n comma-separated query parameter, and always keys the response by\n event KSUID.\n\nUp to 100 IDs may be submitted per request.\n\n## Metrics, review, regression (`/v3/functions/{metrics,review,regression,compare}`)\n\nRoll evaluation results and user corrections up into actionable\nfunction-level signal:\n\n- **`GET /v3/functions/metrics`** — aggregate accuracy, precision,\n recall, F1, and confusion-matrix counts per function.\n- **`POST /v3/functions/review`** — sample-size estimation,\n confidence-bucketed distribution, PR-AUC, and per-threshold\n confidence intervals (Wald or Wilson) for picking review cutoffs.\n- **`POST /v3/functions/regression`** — replay corrected historical\n inputs against a new function version, producing a labeled\n regression dataset.\n- **`POST /v3/functions/regression/corrections`** — propagate\n baseline corrections onto the regression dataset so it can be\n scored.\n- **`POST /v3/functions/compare`** — compute aggregate and\n field-level lift between any two versions, optionally scoped to\n the regression dataset.\n\nAll five endpoints support `extract` end-to-end on both the vision\nand OCR paths, alongside the legacy `transform` / `analyze` / `join`\ntypes."
paths:
/v3/eval:
post:
operationId: v3-trigger-transformation-evaluations
summary: Trigger Transformation Evaluations
description: '**Queue evaluation jobs for a batch of transformations.**
Evaluations run asynchronously and score each transformation''s output
against the function''s schema for confidence, hallucination detection,
and relevance. Transformations must belong to events of a supported
type: `extract`, `transform`, `analyze`, or `join`.
Returns immediately with a summary of queued vs. skipped transformations
and per-transformation errors. Poll `GET /v3/eval/results` to retrieve
results once evaluations complete.'
parameters: []
responses:
'202':
description: The request has been accepted for processing, but processing has not yet completed.
content:
application/json:
schema:
$ref: '#/components/schemas/TriggerEvaluationsResponseV3'
examples:
One transformation rejected for unsupported event type:
summary: One transformation rejected for unsupported event type
value:
queued: 1
skipped: 0
errors:
tr_01HXCD...: 'unsupported event type for evaluation: split. Only ''analyze'', ''transform'', ''extract'', and ''join'' are supported'
Both transformations queued:
summary: Both transformations queued
value:
queued: 2
skipped: 0
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/TriggerEvaluationsRequestV3'
examples:
Queue evaluations for two transformations:
summary: Queue evaluations for two transformations
value:
transformationIDs:
- tr_01HXAB...
- tr_01HXCD...
evaluationVersion: 0.1.0-gemini
/v3/eval/results:
get:
operationId: v3-get-evaluation-results
summary: Get Evaluation Results
description: '**Fetch evaluation results for a batch of events.**
Pass either `eventIDs` (preferred — the externally-stable V3
identifier) or `transformationIDs` as a comma-separated query
parameter. Exactly one of the two must be provided. Up to 100 IDs per
request.
For each requested ID the response reports one of three states: a
completed `result`, still-`pending`, or `failed`. Results, pending,
and failed entries are all keyed by event KSUID regardless of which
input form was used.'
parameters:
- name: eventIDs
in: query
required: false
description: 'Comma-separated list of event KSUIDs to fetch results for. Between
1 and 100 IDs per request. Mutually exclusive with
`transformationIDs`.'
schema:
type: string
explode: false
- name: transformationIDs
in: query
required: false
description: 'Comma-separated list of transformation IDs to fetch results for.
Between 1 and 100 IDs per request. Mutually exclusive with
`eventIDs`. Prefer `eventIDs` for new integrations.'
schema:
type: string
explode: false
- name: evaluationVersion
in: query
required: false
description: Optional evaluation version filter.
schema:
type: string
explode: false
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/EvaluationResultsResponseV3'
examples:
One completed, one still pending:
summary: One completed, one still pending
value:
results:
evt_01HXAB...:
fieldMetrics:
/invoice/number:
confidenceScore: 0.97
reasoning: Matches canonical invoice number in the source document.
hallucination: false
relevanceScore: 1
overallConfidence: 0.97
runtime: 3.42
hasHallucinations: false
evaluationVersion: 0.1.0-gemini
createdAt: '2026-04-23T18:05:00Z'
pending:
- eventID: evt_01HXCD...
createdAt: '2026-04-23T18:04:55Z'
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
/v3/eval/score:
post:
operationId: v3-eval-score-create
summary: Score Function Against (input, expected) Pairs
description: '**Score a function against a list of (input, expected) pairs.**
Submits a batch of `(input, expected)` pairs, runs the named function
over each input, and returns per-pair + aggregate accuracy metrics
comparing the function''s actual output to the provided expected JSON.
Scoring runs asynchronously. The response carries a `scoreRunID`;
poll `GET /v3/eval/score/{scoreRunID}` until `status` is one of
`completed`, `error`, or `cancelled`.
`matchConfig` controls comparator behavior:
- `numericTolerance`: relative tolerance for numeric fields (0 = exact)
- `stringMatch`: `exact` (default) or `fuzzy` (Levenshtein ratio)
- `arrayMatch`: `by-index` (default; only mode in P0)
- `ignorePaths`: JSON Pointer paths to skip, supports `*` wildcards'
parameters: []
responses:
'202':
description: The request has been accepted for processing, but processing has not yet completed.
content:
application/json:
schema:
$ref: '#/components/schemas/EvalScoreCreateResponseV3'
examples:
Score one PDF against expected values:
summary: Score one PDF against expected values
value:
scoreRunID: evalrun_2a8f...
status: pending
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/EvalScoreRequestV3'
/v3/eval/score/{scoreRunID}:
get:
operationId: v3-eval-score-get
summary: Get Score Run
description: '**Get the status and per-pair results of a score run.**
Returns `aggregate` only once `status` reaches `completed`. `perPair`
is populated incrementally — each pair''s `fieldResults` appears as
its underlying function call terminates.'
parameters:
- name: scoreRunID
in: path
required: true
description: The `scoreRunID` returned by `POST /v3/eval/score`.
schema:
type: string
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/EvalScoreRunResponseV3'
examples:
Completed run with a perfect score:
summary: Completed run with a perfect score
value:
scoreRunID: evalrun_2a8f...
status: completed
functionName: invoice-extractor
functionVersionNum: 3
matchConfig:
numericTolerance: 0.01
stringMatch: exact
progress:
total: 1
completed: 1
failed: 0
aggregate:
precision: 1
recall: 1
f1: 1
exactMatches: 2
withinTolerance: 0
fuzzyMatches: 0
misses: 0
extras: 0
totalFieldsExpected: 2
totalFieldsActual: 2
perPair:
- pairIndex: 0
callID: call_xxx
status: completed
fieldResults:
- path: /invoiceNumber
match: exact
- path: /total
match: exact
'404':
description: The server cannot find the requested resource.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
/v3/eval/score/{scoreRunID}/cancel:
post:
operationId: v3-eval-score-cancel
summary: Cancel Score Run
description: '**Cancel an in-flight score run.**
Transitions the run to `cancelled`. Function calls already in flight
are allowed to finish (best-effort cancellation via the job queue);
results from completed pairs may still appear in subsequent GETs.'
parameters:
- name: scoreRunID
in: path
required: true
description: The `scoreRunID` returned by `POST /v3/eval/score`.
schema:
type: string
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/EvalScoreRunResponseV3'
'404':
description: The server cannot find the requested resource.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
/v3/functions/compare:
post:
operationId: v3-function-version-compare
summary: Compare Metrics Between Function Versions
description: '**Compare metrics between two function versions.**
Computes aggregate and field-level lift/regression between any two
versions of a function: accuracy, precision, recall, F1, and PR-AUC.
Field-level changes are returned only for fields whose lift exceeds
1% in either direction.
Supported for every function type that produces labeled
transformations: `extract`, `transform`, `analyze`, `join`. Pass
`isRegression: true` to compare only the regression dataset (rows
produced by `POST /v3/functions/regression`) — the canonical way to
judge a candidate version before promoting it.
Defaults: `baselineVersionNum = currentVersionNum - 1`,
`comparisonVersionNum = currentVersionNum`.'
parameters: []
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/functionVersionCompareResponse'
examples:
Function improved:
summary: Function improved
value:
functionName: invoice-extractor
baselineVersionNum: 2
comparisonVersionNum: 3
baselineTransformationCount: 100
comparisonTransformationCount: 150
aggregateComparison:
accuracy:
baselineValue: 0.85
comparisonValue: 0.92
difference: 0.07
liftPercent: 8.24
f1Score:
baselineValue: 0.85
comparisonValue: 0.92
difference: 0.07
liftPercent: 8.24
fieldMetricsChanges: []
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
'404':
description: The server cannot find the requested resource.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
'500':
description: Server error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/functionVersionCompareRequest'
examples:
Compare versions on the regression dataset only:
summary: Compare versions on the regression dataset only
value:
functionName: invoice-extractor
baselineVersionNum: 2
comparisonVersionNum: 3
isRegression: true
Compare two versions on labeled data:
summary: Compare two versions on labeled data
value:
functionName: invoice-extractor
baselineVersionNum: 2
comparisonVersionNum: 3
/v3/functions/metrics:
get:
operationId: v3-get-function-metrics
summary: Get Function Metrics
description: '**Retrieve performance metrics for functions based on labeled transformation data.**
Calculates accuracy, precision, recall, F1, and the underlying confusion-matrix
counts for each matching function by comparing model outputs against user
corrections. Metrics are aggregated across every transformation the function
has produced, regardless of function type — `extract`, `transform`, `analyze`,
and `join` all populate the same `metrics` column on the transformation row,
so v3 surfaces all of them uniformly.
## Filtering
Combine `functionIDs` / `functionNames` / `types` to narrow the result set.
`types` accepts `extract` alongside the legacy `transform` / `analyze` types
(which remain readable). Pagination is cursor-based.
## Requirements
A function only shows non-zero metrics once at least one of its transformations
has been labeled — submit corrections via `POST /v3/events/{eventID}/feedback`.'
parameters:
- name: limit
in: query
required: false
schema:
type: integer
minimum: 1
maximum: 100
default: 50
- name: functionIDs
in: query
required: false
schema:
type: array
items:
type: string
minItems: 1
explode: false
- name: functionNames
in: query
required: false
schema:
type: array
items:
type: string
minItems: 1
explode: false
- name: types
in: query
required: false
schema:
type: array
items:
$ref: '#/components/schemas/FunctionType'
minItems: 1
explode: false
- name: sortOrder
in: query
required: false
description: 'Sort direction over the result set (default `asc`). Pagination works
symmetrically in both directions via `startingAfter` / `endingBefore`.'
schema:
type: string
enum:
- asc
- desc
default: asc
- name: startingAfter
in: query
required: false
description: Cursor — a `functionID` defining your place in the list.
schema:
type: string
- name: endingBefore
in: query
required: false
description: Cursor — a `functionID` defining your place in the list.
schema:
type: string
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/FunctionMetricsResponse'
example:
functions:
- functionName: invoice-extractor
totalLabeledResults: 54
totalResults: 120
metrics:
accuracy: 0.7407
precision: 0.9524
recall: 0.7692
f1Score: 0.8511
tp: 40
fp: 2
tn: 0
fn: 12
totalCount: 1
tags:
- Function Accuracy
/v3/functions/regression:
post:
operationId: v3-function-regression
summary: Run Function Regression Testing
description: '**Kick off a regression run between two versions of a function.**
Replays a sample of corrected historical inputs against the comparison
version, producing fresh transformations marked `isRegression: true`.
Each new run returns the workflow `callID`s you can monitor via
`GET /v3/calls/{callID}`.
Supported for every function type that produces correctable
transformations: `extract`, `transform`, `analyze`, `join`. For
`extract` specifically, the regression sample is dispatched through
the same OCR vs. vision path used at original call time (PDF, PNG,
JPEG, HEIC, HEIF, WebP go through the vision worker; everything else
goes through OCR → transform).
The comparison version must share a schema-compatible output shape
with the baseline; structural differences are reported as a 400 with
the offending field-level diffs.
## Typical flow
1. `POST /v3/functions/regression` — queues calls, returns
`{ originalReferenceID, callID }` per sample.
2. Wait (poll `GET /v3/calls/{callID}` or subscribe to webhooks).
3. `POST /v3/functions/regression/corrections` to copy baseline
corrections onto the new regression transformations.
4. `POST /v3/functions/compare` to compare baseline vs comparison
metrics for the regression dataset.'
parameters: []
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/functionRegressionResponse'
examples:
Regression run queued:
summary: Regression run queued
description: Calls are processing asynchronously — poll the calls API or wait on webhooks.
value:
functionName: invoice-extractor
result:
functionName: invoice-extractor
totalSamples: 50
calls:
- originalReferenceID: invoice-123
callID: wc_2N6gH8ZKCmvb6BnFcGqhKJ98VzP
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
'404':
description: The server cannot find the requested resource.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/functionRegressionRequest'
examples:
Compare specific versions:
summary: Compare specific versions
value:
functionName: invoice-extractor
baselineVersionNum: 3
comparisonVersionNum: 5
sampleSize: 100
onlyCorrectedData: true
Default regression (previous version → current version):
summary: Default regression (previous version → current version)
value:
functionName: invoice-extractor
sampleSize: 50
/v3/functions/regression/corrections:
post:
operationId: v3-apply-baseline-corrections
summary: Apply Baseline Corrections to Regression Transformations
description: '**Copy baseline corrections onto regression transformations.**
Looks up regression transformations created against the comparison
version (`isRegression: true`, `correctedJSON IS NULL`), finds the
matching baseline transformation by `referenceID`, and copies the
baseline''s `correctedJSON` onto the regression row via the same code
path used by `POST /v3/events/{eventID}/feedback`. The applied
corrections are immediately scored against the regression output,
populating the confusion-matrix metrics used by `function-review` and
`function-version-compare`.
Works for every function type that produces correctable
transformations, including `extract` on both the vision and OCR
paths. (Previously the vision path silently dropped `is_regression`
during the original regression run, so no rows matched the
predicate — that has been fixed.)
Returns counts plus the list of **event KSUIDs** whose underlying
regression transformation received a correction. Errors (e.g.
baseline transformation missing for a given `referenceID`) are
returned per-row in the `errors` map, keyed by event KSUID, rather
than aborting the whole call.'
parameters: []
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/applyBaselineCorrectionsResponseV3'
examples:
Partial application with per-row errors:
summary: Partial application with per-row errors
value:
applied: 18
appliedEventIDs:
- evt_2N6gH8...
- evt_2N6gH9...
skipped: 2
errors:
evt_2N6gHB...: 'baseline transformation not found for reference ID: invoice-789'
All corrections applied:
summary: All corrections applied
value:
applied: 25
appliedEventIDs:
- evt_2N6gH8...
- evt_2N6gH9...
- evt_2N6gHA...
skipped: 0
errors: {}
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
'404':
description: The server cannot find the requested resource.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
'500':
description: Server error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/applyBaselineCorrectionsRequest'
examples:
Apply corrections from v3 onto v4 regression set:
summary: Apply corrections from v3 onto v4 regression set
value:
functionName: invoice-extractor
baselineVersionNum: 3
comparisonVersionNum: 4
/v3/functions/review:
post:
operationId: v3-function-review
summary: Function Review
description: '**Estimate human review requirements for a function.**
Combines confusion-matrix metrics with the per-transformation evaluation
scores (confidence / hallucination / relevance produced by the eval service)
to compute:
- A confidence-bucketed distribution of the function''s outputs.
- Sample-size estimates at configurable margin-of-error and confidence
levels (Wald or Wilson intervals).
- A precision-recall AUC and a per-threshold matrix you can use to
pick a review cutoff.
Supported for every function type that produces transformations and feeds
the auto-evaluation pipeline: `extract`, `transform`, `analyze`, `join`.
Extract works on both vision (PDF/PNG/JPEG/HEIC/HEIF/WebP) and OCR-routed
inputs.
Pass `isRegression: true` to scope the review to transformations created
by a previous regression run (see `POST /v3/functions/regression`).'
parameters: []
responses:
'200':
description: The request has succeeded.
content:
application/json:
schema:
$ref: '#/components/schemas/FunctionReviewResponse'
example:
functionName: invoice-extractor
functionVersionNum: 3
estimate:
totalTransformations: 1000
labeledTransformations: 200
unlabeledTransformations: 800
missingEvaluations: 50
confidenceDistribution:
high: 500
medium: 350
low: 150
thresholdMatrix:
- threshold: 0.8
tp: 85
fp: 12
fn: 15
tn: 88
accuracyAboveThreshold:
'95':
ciLower: 0.8456
mid: 0.875
ciUpper: 0.9044
currentSample: 120
sampleNeeded: 30
metrics:
fieldMetrics:
- fieldPath: /invoice/number
metrics:
accuracy: 0.95
precision: 0.98
recall: 0.92
f1Score: 0.95
tp: 92
fp: 2
tn: 0
fn: 8
precisionRecallAuc: 0.8542
aggregateMetrics:
accuracy: 0.7407
precision: 0.9524
recall: 0.7692
f1Score: 0.8511
tp: 40
fp: 2
tn: 0
fn: 12
'400':
description: The server could not understand the request due to invalid syntax.
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
'500':
description: Server error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPError'
tags:
- Function Accuracy
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/FunctionReviewRequest'
examples:
Review the regression dataset for a specific version:
summary: Review the regression dataset for a specific version
value:
functionName: invoice-extractor
functionVersionNum: 2
isRegression: true
marginOfError: 0.05
Full threshold analysis (default):
summary: Full threshold analysis (default)
value:
functionName: invoice-extractor
marginOfError: 0.05
thresholdMin: 0.5
thresholdMax: 1
thresholdStep: 0.01
components:
schemas:
EvaluationResultsResponseV3:
type
# --- truncated at 32 KB (71 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/bem/refs/heads/main/openapi/bem-function-accuracy-api-openapi.yml