Feldera Pipeline CRUD API
The Pipeline CRUD API from Feldera — 2 operation(s) for pipeline crud.
OpenAPI Specification
openapi: 3.0.3
info:
title: Feldera Input Connectors Pipeline CRUD API
description: "\nWith Feldera, users create data pipelines out of SQL programs.\nA SQL program comprises tables and views, and includes as well the definition of\ninput and output connectors for each respectively. A connector defines a data\nsource or data sink to feed input data into tables or receive output data\ncomputed by the views respectively.\n\n## Pipeline\n\nThe API is centered around the **pipeline**, which most importantly consists\nout of the SQL program, but also has accompanying metadata and configuration parameters\n(e.g., compilation profile, number of workers, etc.).\n\n* A pipeline is identified and referred to by its user-provided unique name.\n* The pipeline program is asynchronously compiled when the pipeline is first created or\n when its program is subsequently updated.\n* Pipeline deployment start is only able to proceed to provisioning once the program is successfully\n compiled.\n* A pipeline cannot be updated while it is deployed.\n\n## Concurrency\n\nEach pipeline has a version, which is incremented each time its core fields are updated.\nThe version is monotonically increasing. There is additionally a program version which covers\nonly the program-related core fields, and is used by the compiler to discern when to recompile.\n\n## Client request handling\n\n### Request outcome expectations\n\nThe outcome of a request is that it either fails (e.g., DNS lookup failed) without any response\n(no status code nor body), or it succeeds and gets back a response status code and body.\n\nIn case of a response, usually it is the Feldera endpoint that generated it:\n- If it is success (2xx), it will return whichever body belongs to the success response.\n- Otherwise, if it is an error (4xx, 5xx), it will return a Feldera error response JSON body\n which will have an application-level `error_code`.\n\nHowever, there are two notable exceptions when the response is not generated by the Feldera\nendpoint:\n- If the HTTP server, to which the endpoint belongs, encountered an issue, it might return\n 4xx (e.g., for an unknown endpoint) or 5xx error codes by itself (e.g., when it is initializing).\n- If the Feldera API server is behind a (reverse) proxy, the proxy can return error codes by itself,\n for example BAD GATEWAY (502) or GATEWAY TIMEOUT (504).\n\nAs such, it is not guaranteed that the (4xx, 5xx) will have a Feldera error response JSON body\nin these latter cases.\n\n### Error handling and retrying\n\nThe error type returned by the client should distinguish between the error responses generated\nby Feldera endpoints themselves (which have a Feldera error response body) and those that are\ngenerated by other sources.\n\nIn order for a client operation (e.g., `pipeline.resume()`) to be robust (i.e., not fail due to\na single HTTP request not succeeding) the client should use a retry mechanism if the operation\nis idempotent. The retry mechanism must however have a time limit, after which it times out.\nThis guarantees that the client operation is eventually responsive, which enables the script\nit is a part of to not hang indefinitely on Feldera operations and instead be able to decide\nby itself whether and how to proceed. If no response is returned, the mechanism should generally\nretry. When a response is returned, the decision whether to retry can generally depend on the status\ncode: especially the status codes 408, 502, 503 and 504 should be considered as transient errors.\nFiner grained retry decisions should be made by taking into account the application-level\n`error_code` if the response body was indeed a Feldera error response body.\n\n## Feldera client errors (4xx)\n\n_Client behavior:_ clients should generally return with an error when they get back a 4xx status\ncode, as it usually means the request will likely not succeed even if it is sent again. Certain\nrequests might make use of a timed retry mechanism when the client error is transient without\nrequiring any user intervention to overcome, for instance a transaction already being in progress\nleading to a temporary CONFLICT (409) error.\n\n- **BAD REQUEST (400)**: invalid user request (general).\n - _Example:_ the new pipeline name `example1@~` contains invalid characters.\n\n- **UNAUTHORIZED (401)**: the user is not authorized to issue the request.\n - _Example:_ an invalid API key is provided.\n\n- **NOT FOUND (404)**: a resource required to exist in order to process the request was not found.\n - _Example:_ a pipeline named `example` does not exist when trying to update it.\n\n- **CONFLICT (409)**: there is a conflict between the request and a relevant resource.\n - _Example:_ a pipeline named `example` already exists.\n - _Example:_ another transaction is already in process.\n\n## Feldera server errors (5xx)\n\n- **INTERNAL SERVER ERROR (500)**: the server is unexpectedly unable to process the request\n (general).\n - _Example:_ unable to reach the database.\n - _Client behavior:_ immediately return with an error.\n\n- **NOT IMPLEMENTED (501)**: the server does not implement functionality required to process the\n request.\n - _Example:_ making a request to an enterprise-only endpoint in the OSS edition.\n - _Client behavior:_ immediately return with an error.\n\n- **SERVICE UNAVAILABLE (503)**: the server is not (yet) able to process the request.\n - _Example:_ pausing a pipeline which is still provisioning.\n - _Client behavior:_ depending on the type of request, client may use a timed retry mechanism.\n\n## Feldera error response body\n\nWhen the Feldera API returns an HTTP error status code (4xx, 5xx), the body will contain the\nfollowing JSON object:\n\n```json\n{\n \"message\": \"Human-readable explanation.\",\n \"error_code\": \"CodeSpecifyingError\",\n \"details\": {\n\n }\n}\n```\n\nIt contains the following fields:\n- **message (string)**: human-readable explanation of the error that occurred and potentially\n hinting what can be done about it.\n- **error_code (string)**: application-level code about the error that occurred, written in CamelCase.\n For example: `UnknownPipelineName`, `DuplicateName`, `PauseWhileNotProvisioned`, ... .\n- **details (object)**: JSON object corresponding to the `error_code` with fields that provide\n details relevant to it. For example: if a name is unknown, a field with the unknown name in\n question.\n"
contact:
name: Feldera Team
email: dev@feldera.com
license:
name: MIT OR Apache-2.0
version: 0.323.0
tags:
- name: Pipeline CRUD
paths:
/v0/pipelines:
get:
tags:
- Pipeline CRUD
summary: List Pipelines
description: 'Retrieve the list of pipelines.
Configure which fields are included using the `selector` query parameter.'
operationId: list_pipelines
parameters:
- name: selector
in: query
description: 'The `selector` parameter limits which fields are returned for a pipeline.
Limiting which fields is particularly handy for instance when frequently
monitoring over low bandwidth connections while being only interested
in pipeline status.'
required: false
schema:
$ref: '#/components/schemas/PipelineFieldSelector'
responses:
'200':
description: List of pipelines retrieved successfully
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/PipelineSelectedInfo'
example:
- id: 67e55044-10b1-426f-9247-bb680e5fe0c8
name: example1
description: Description of the pipeline example1
tags: []
created_at: '1970-01-01T00:00:00Z'
version: 4
platform_version: v0
runtime_config:
workers: 16
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: true
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 0
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: null
memory_mb_max: null
storage_mb_max: null
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 1000000
pin_cpus: []
provisioning_timeout_secs: null
max_parallel_connector_init: null
init_containers: null
checkpoint_during_suspend: true
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE table1 ( col1 INT );
udf_rust: ''
udf_toml: ''
program_config:
profile: optimized
cache: true
runtime_version: null
use_platform_compiler: false
program_version: 2
program_status: Pending
program_status_since: '1970-01-01T00:00:00Z'
program_error:
sql_compilation: null
rust_compilation: null
system_error: null
program_info: null
deployment_error: null
refresh_version: 4
storage_status: Cleared
storage_status_details: null
deployment_id: null
deployment_initial: null
deployment_status: Stopped
deployment_status_since: '1970-01-01T00:00:00Z'
deployment_desired_status: Stopped
deployment_desired_status_since: '1970-01-01T00:00:00Z'
deployment_resources_status: Stopped
deployment_resources_status_details: null
deployment_resources_status_since: '1970-01-01T00:00:00Z'
deployment_resources_desired_status: Stopped
deployment_resources_desired_status_since: '1970-01-01T00:00:00Z'
deployment_runtime_status: null
deployment_runtime_status_details: null
deployment_runtime_status_since: null
deployment_runtime_desired_status: null
deployment_runtime_desired_status_since: null
- id: 67e55044-10b1-426f-9247-bb680e5fe0c9
name: example2
description: Description of the pipeline example2
tags: []
created_at: '1970-01-01T00:00:00Z'
version: 1
platform_version: v0
runtime_config:
workers: 10
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: false
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 100000
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: 1000
memory_mb_max: null
storage_mb_max: 10000
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 100000
pin_cpus: []
provisioning_timeout_secs: 1200
max_parallel_connector_init: 10
init_containers: null
checkpoint_during_suspend: false
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE table2 ( col2 VARCHAR );
udf_rust: ''
udf_toml: ''
program_config:
profile: unoptimized
cache: true
runtime_version: null
use_platform_compiler: false
program_version: 1
program_status: Pending
program_status_since: '1970-01-01T00:00:00Z'
program_error:
sql_compilation: null
rust_compilation: null
system_error: null
program_info: null
deployment_error: null
refresh_version: 1
storage_status: Cleared
storage_status_details: null
deployment_id: null
deployment_initial: null
deployment_status: Stopped
deployment_status_since: '1970-01-01T00:00:00Z'
deployment_desired_status: Stopped
deployment_desired_status_since: '1970-01-01T00:00:00Z'
deployment_resources_status: Stopped
deployment_resources_status_details: null
deployment_resources_status_since: '1970-01-01T00:00:00Z'
deployment_resources_desired_status: Stopped
deployment_resources_desired_status_since: '1970-01-01T00:00:00Z'
deployment_runtime_status: null
deployment_runtime_status_details: null
deployment_runtime_status_since: null
deployment_runtime_desired_status: null
deployment_runtime_desired_status_since: null
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
security:
- JSON web token (JWT) or API key: []
post:
tags:
- Pipeline CRUD
summary: Create Pipeline
description: Create a new pipeline with the provided configuration.
operationId: post_pipeline
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/PostPutPipeline'
example:
name: example1
description: Description of the pipeline example1
tags: []
runtime_config:
workers: 16
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: true
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 0
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: null
memory_mb_max: null
storage_mb_max: null
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 1000000
pin_cpus: []
provisioning_timeout_secs: null
max_parallel_connector_init: null
init_containers: null
checkpoint_during_suspend: true
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE table1 ( col1 INT );
udf_rust: null
udf_toml: null
program_config:
profile: optimized
cache: true
runtime_version: null
use_platform_compiler: false
required: true
responses:
'201':
description: Pipeline successfully created
content:
application/json:
schema:
$ref: '#/components/schemas/PipelineInfo'
example:
id: 67e55044-10b1-426f-9247-bb680e5fe0c8
name: example1
description: Description of the pipeline example1
tags: []
created_at: '1970-01-01T00:00:00Z'
version: 4
platform_version: v0
runtime_config:
workers: 16
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: true
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 0
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: null
memory_mb_max: null
storage_mb_max: null
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 1000000
pin_cpus: []
provisioning_timeout_secs: null
max_parallel_connector_init: null
init_containers: null
checkpoint_during_suspend: true
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE table1 ( col1 INT );
udf_rust: ''
udf_toml: ''
program_config:
profile: optimized
cache: true
runtime_version: null
use_platform_compiler: false
program_version: 2
program_status: Pending
program_status_since: '1970-01-01T00:00:00Z'
program_error:
sql_compilation: null
rust_compilation: null
system_error: null
program_info: null
deployment_error: null
refresh_version: 4
storage_status: Cleared
storage_status_details: null
deployment_id: null
deployment_initial: null
deployment_status: Stopped
deployment_status_since: '1970-01-01T00:00:00Z'
deployment_desired_status: Stopped
deployment_desired_status_since: '1970-01-01T00:00:00Z'
deployment_resources_status: Stopped
deployment_resources_status_details: null
deployment_resources_status_since: '1970-01-01T00:00:00Z'
deployment_resources_desired_status: Stopped
deployment_resources_desired_status_since: '1970-01-01T00:00:00Z'
deployment_runtime_status: null
deployment_runtime_status_details: null
deployment_runtime_status_since: null
deployment_runtime_desired_status: null
deployment_runtime_desired_status_since: null
'400':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Name does not match pattern:
value:
message: 'Name ''name-with-invalid-char-#'' should be non-empty and only contain lowercase (a-z), uppercase (A-Z), number (0-9), underscore (_) or hyphen (-) characters, but cannot start or end with hyphen or underscore (pattern: ''^([A-Za-z0-9][-A-Za-z0-9_]*)?[A-Za-z0-9]$'')'
error_code: NameDoesNotMatchPattern
details:
name: name-with-invalid-char-#
pattern: ^([A-Za-z0-9][-A-Za-z0-9_]*)?[A-Za-z0-9]$
pattern_description: be non-empty and only contain lowercase (a-z), uppercase (A-Z), number (0-9), underscore (_) or hyphen (-) characters, but cannot start or end with hyphen or underscore
'409':
description: Cannot create pipeline as the name already exists
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: An entity with this name already exists
error_code: DuplicateName
details: null
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}:
get:
tags:
- Pipeline CRUD
summary: Get Pipeline
description: 'Retrieve a pipeline.
Configure which fields are included using the `selector` query parameter.'
operationId: get_pipeline
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: selector
in: query
description: 'The `selector` parameter limits which fields are returned for a pipeline.
Limiting which fields is particularly handy for instance when frequently
monitoring over low bandwidth connections while being only interested
in pipeline status.'
required: false
schema:
$ref: '#/components/schemas/PipelineFieldSelector'
responses:
'200':
description: Pipeline retrieved successfully
content:
application/json:
schema:
$ref: '#/components/schemas/PipelineSelectedInfo'
example:
id: 67e55044-10b1-426f-9247-bb680e5fe0c8
name: example1
description: Description of the pipeline example1
tags: []
created_at: '1970-01-01T00:00:00Z'
version: 4
platform_version: v0
runtime_config:
workers: 16
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: true
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 0
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: null
memory_mb_max: null
storage_mb_max: null
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 1000000
pin_cpus: []
provisioning_timeout_secs: null
max_parallel_connector_init: null
init_containers: null
checkpoint_during_suspend: true
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE table1 ( col1 INT );
udf_rust: ''
udf_toml: ''
program_config:
profile: optimized
cache: true
runtime_version: null
use_platform_compiler: false
program_version: 2
program_status: Pending
program_status_since: '1970-01-01T00:00:00Z'
program_error:
sql_compilation: null
rust_compilation: null
system_error: null
program_info: null
deployment_error: null
refresh_version: 4
storage_status: Cleared
storage_status_details: null
deployment_id: null
deployment_initial: null
deployment_status: Stopped
deployment_status_since: '1970-01-01T00:00:00Z'
deployment_desired_status: Stopped
deployment_desired_status_since: '1970-01-01T00:00:00Z'
deployment_resources_status: Stopped
deployment_resources_status_details: null
deployment_resources_status_since: '1970-01-01T00:00:00Z'
deployment_resources_desired_status: Stopped
deployment_resources_desired_status_since: '1970-01-01T00:00:00Z'
deployment_runtime_status: null
deployment_runtime_status_details: null
deployment_runtime_status_since: null
deployment_runtime_desired_status: null
deployment_runtime_desired_status_since: null
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
security:
- JSON web token (JWT) or API key: []
put:
tags:
- Pipeline CRUD
summary: Upsert Pipeline
description: Fully update a pipeline if it already exists, otherwise create a new pipeline.
operationId: put_pipeline
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/PostPutPipeline'
example:
name: example1
description: Description of the pipeline example1
tags: []
runtime_config:
workers: 16
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: true
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 0
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: null
memory_mb_max: null
storage_mb_max: null
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 1000000
pin_cpus: []
provisioning_timeout_secs: null
max_parallel_connector_init: null
init_containers: null
checkpoint_during_suspend: true
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE table1 ( col1 INT );
udf_rust: null
udf_toml: null
program_config:
profile: optimized
cache: true
runtime_version: null
use_platform_compiler: false
required: true
responses:
'200':
description: Pipeline successfully updated
content:
application/json:
schema:
$ref: '#/components/schemas/PipelineInfo'
example:
id: 67e55044-10b1-426f-9247-bb680e5fe0c8
name: example1
description: Description of the pipeline example1
tags: []
created_at: '1970-01-01T00:00:00Z'
version: 4
platform_version: v0
runtime_config:
workers: 16
max_rss_mb: null
datafusion_memory_mb: null
hosts: 1
storage:
backend:
name: default
min_storage_bytes: null
min_step_storage_bytes: null
compression: default
cache_mib: null
fault_tolerance:
model: none
checkpoint_interval_secs: 60
cpu_profiler: true
tracing: false
tracing_endpoint_jaeger: ''
min_batch_size_records: 0
max_buffering_delay_usecs: 0
resources:
cpu_cores_min: null
cpu_cores_max: null
memory_mb_min: null
memory_mb_max: null
storage_mb_max: null
storage_class: null
service_account_name: null
namespace: null
clock_resolution_usecs: 1000000
pin_cpus: []
provisioning_timeout_secs: null
max_parallel_connector_init: null
init_containers: null
checkpoint_during_suspend: true
http_workers: null
io_workers: null
env: {}
dev_tweaks: {}
logging: null
pipeline_template_configmap: null
program_code: CREATE TABLE
# --- truncated at 32 KB (230 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/feldera/refs/heads/main/openapi/feldera-pipeline-crud-api-openapi.yml