Feldera Pipeline Lifecycle API
The Pipeline Lifecycle API from Feldera — 20 operation(s) for pipeline lifecycle.
The Pipeline Lifecycle API from Feldera — 20 operation(s) for pipeline lifecycle.
openapi: 3.0.3
info:
title: Feldera Input Connectors Pipeline Lifecycle API
description: "\nWith Feldera, users create data pipelines out of SQL programs.\nA SQL program comprises tables and views, and includes as well the definition of\ninput and output connectors for each respectively. A connector defines a data\nsource or data sink to feed input data into tables or receive output data\ncomputed by the views respectively.\n\n## Pipeline\n\nThe API is centered around the **pipeline**, which most importantly consists\nout of the SQL program, but also has accompanying metadata and configuration parameters\n(e.g., compilation profile, number of workers, etc.).\n\n* A pipeline is identified and referred to by its user-provided unique name.\n* The pipeline program is asynchronously compiled when the pipeline is first created or\n when its program is subsequently updated.\n* Pipeline deployment start is only able to proceed to provisioning once the program is successfully\n compiled.\n* A pipeline cannot be updated while it is deployed.\n\n## Concurrency\n\nEach pipeline has a version, which is incremented each time its core fields are updated.\nThe version is monotonically increasing. There is additionally a program version which covers\nonly the program-related core fields, and is used by the compiler to discern when to recompile.\n\n## Client request handling\n\n### Request outcome expectations\n\nThe outcome of a request is that it either fails (e.g., DNS lookup failed) without any response\n(no status code nor body), or it succeeds and gets back a response status code and body.\n\nIn case of a response, usually it is the Feldera endpoint that generated it:\n- If it is success (2xx), it will return whichever body belongs to the success response.\n- Otherwise, if it is an error (4xx, 5xx), it will return a Feldera error response JSON body\n which will have an application-level `error_code`.\n\nHowever, there are two notable exceptions when the response is not generated by the Feldera\nendpoint:\n- If the HTTP server, to which the endpoint belongs, encountered an issue, it might return\n 4xx (e.g., for an unknown endpoint) or 5xx error codes by itself (e.g., when it is initializing).\n- If the Feldera API server is behind a (reverse) proxy, the proxy can return error codes by itself,\n for example BAD GATEWAY (502) or GATEWAY TIMEOUT (504).\n\nAs such, it is not guaranteed that the (4xx, 5xx) will have a Feldera error response JSON body\nin these latter cases.\n\n### Error handling and retrying\n\nThe error type returned by the client should distinguish between the error responses generated\nby Feldera endpoints themselves (which have a Feldera error response body) and those that are\ngenerated by other sources.\n\nIn order for a client operation (e.g., `pipeline.resume()`) to be robust (i.e., not fail due to\na single HTTP request not succeeding) the client should use a retry mechanism if the operation\nis idempotent. The retry mechanism must however have a time limit, after which it times out.\nThis guarantees that the client operation is eventually responsive, which enables the script\nit is a part of to not hang indefinitely on Feldera operations and instead be able to decide\nby itself whether and how to proceed. If no response is returned, the mechanism should generally\nretry. When a response is returned, the decision whether to retry can generally depend on the status\ncode: especially the status codes 408, 502, 503 and 504 should be considered as transient errors.\nFiner grained retry decisions should be made by taking into account the application-level\n`error_code` if the response body was indeed a Feldera error response body.\n\n## Feldera client errors (4xx)\n\n_Client behavior:_ clients should generally return with an error when they get back a 4xx status\ncode, as it usually means the request will likely not succeed even if it is sent again. Certain\nrequests might make use of a timed retry mechanism when the client error is transient without\nrequiring any user intervention to overcome, for instance a transaction already being in progress\nleading to a temporary CONFLICT (409) error.\n\n- **BAD REQUEST (400)**: invalid user request (general).\n - _Example:_ the new pipeline name `example1@~` contains invalid characters.\n\n- **UNAUTHORIZED (401)**: the user is not authorized to issue the request.\n - _Example:_ an invalid API key is provided.\n\n- **NOT FOUND (404)**: a resource required to exist in order to process the request was not found.\n - _Example:_ a pipeline named `example` does not exist when trying to update it.\n\n- **CONFLICT (409)**: there is a conflict between the request and a relevant resource.\n - _Example:_ a pipeline named `example` already exists.\n - _Example:_ another transaction is already in process.\n\n## Feldera server errors (5xx)\n\n- **INTERNAL SERVER ERROR (500)**: the server is unexpectedly unable to process the request\n (general).\n - _Example:_ unable to reach the database.\n - _Client behavior:_ immediately return with an error.\n\n- **NOT IMPLEMENTED (501)**: the server does not implement functionality required to process the\n request.\n - _Example:_ making a request to an enterprise-only endpoint in the OSS edition.\n - _Client behavior:_ immediately return with an error.\n\n- **SERVICE UNAVAILABLE (503)**: the server is not (yet) able to process the request.\n - _Example:_ pausing a pipeline which is still provisioning.\n - _Client behavior:_ depending on the type of request, client may use a timed retry mechanism.\n\n## Feldera error response body\n\nWhen the Feldera API returns an HTTP error status code (4xx, 5xx), the body will contain the\nfollowing JSON object:\n\n```json\n{\n \"message\": \"Human-readable explanation.\",\n \"error_code\": \"CodeSpecifyingError\",\n \"details\": {\n\n }\n}\n```\n\nIt contains the following fields:\n- **message (string)**: human-readable explanation of the error that occurred and potentially\n hinting what can be done about it.\n- **error_code (string)**: application-level code about the error that occurred, written in CamelCase.\n For example: `UnknownPipelineName`, `DuplicateName`, `PauseWhileNotProvisioned`, ... .\n- **details (object)**: JSON object corresponding to the `error_code` with fields that provide\n details relevant to it. For example: if a name is unknown, a field with the unknown name in\n question.\n"
contact:
name: Feldera Team
email: dev@feldera.com
license:
name: MIT OR Apache-2.0
version: 0.323.0
tags:
- name: Pipeline Lifecycle
paths:
/v0/pipelines/{pipeline_name}/activate:
post:
tags:
- Pipeline Lifecycle
summary: Activate Standby Pipeline
description: 'Requests the pipeline to activate if it is currently in standby mode, which it will do
asynchronously.
Progress should be monitored by polling the pipeline `GET` endpoints.
This endpoint is only applicable when the pipeline is configured to start
from object store and started as standby.'
operationId: post_pipeline_activate
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: initial
in: query
required: false
schema:
$ref: '#/components/schemas/RuntimeDesiredStatus'
responses:
'202':
description: Pipeline activation initiated
content:
application/json:
schema:
type: string
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/approve:
post:
tags:
- Pipeline Lifecycle
summary: Approve Bootstrap
description: 'Approves the pipeline to proceed with bootstrapping.
This endpoint is used when a pipeline has been started with
`bootstrap_policy=await_approval`, it is resuming from an existing checkpoint,
but the pipeline has been modified since the checkpoint was made and is
currently in the `AwaitingApproval` state awaiting user approval to proceed
with bootstrapping.'
operationId: post_pipeline_approve
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: silent_bootstrap
in: query
description: Bootstrap the pipeline with output connectors disabled.
required: false
schema:
type: boolean
- name: concurrent_bootstrap
in: query
description: 'Bootstrap new and modified views concurrently, keeping the pre-existing
views live while the new ones backfill. Mutually exclusive with
`silent_bootstrap`.'
required: false
schema:
type: boolean
responses:
'200':
description: Bootstrap approved
content:
application/json:
schema:
type: string
'400':
description: Bootstrap options are mutually inconsistent
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/checkpoint:
post:
tags:
- Pipeline Lifecycle
summary: Checkpoint Now
description: 'Initiates checkpoint for a running or paused pipeline.
Returns a checkpoint sequence number that can be used with `/checkpoint_status` to
determine when the checkpoint has completed.'
operationId: checkpoint_pipeline
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
responses:
'200':
description: Checkpoint initiated
content:
application/json:
schema:
$ref: '#/components/schemas/CheckpointResponse'
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/checkpoint/sync:
post:
tags:
- Pipeline Lifecycle
summary: Sync Checkpoints To S3
description: Syncs latest checkpoints to the object store configured in pipeline config.
operationId: sync_checkpoint
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
responses:
'202':
description: Checkpoint sync to object store initiated
content:
application/json:
schema:
$ref: '#/components/schemas/CheckpointSyncResponse'
'404':
description: No checkpoints found
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/checkpoint/sync_status:
get:
tags:
- Pipeline Lifecycle
summary: Get Checkpoint Sync Status
description: Retrieve status of checkpoint sync activity in a pipeline.
operationId: get_checkpoint_sync_status
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
responses:
'200':
description: Checkpoint sync status retrieved successfully
content:
application/json:
schema:
$ref: '#/components/schemas/CheckpointSyncStatus'
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/checkpoint_status:
get:
tags:
- Pipeline Lifecycle
summary: Get Checkpoint Status
description: Retrieve status of checkpoint activity in a pipeline.
operationId: get_checkpoint_status
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
responses:
'200':
description: Checkpoint status retrieved successfully
content:
application/json:
schema:
$ref: '#/components/schemas/CheckpointStatus'
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
# --- truncated at 32 KB (266 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/feldera/refs/heads/main/openapi/feldera-pipeline-lifecycle-api-openapi.yml