openapi: 3.0.3
info:
title: Feldera Input Connectors API
description: "\nWith Feldera, users create data pipelines out of SQL programs.\nA SQL program comprises tables and views, and includes as well the definition of\ninput and output connectors for each respectively. A connector defines a data\nsource or data sink to feed input data into tables or receive output data\ncomputed by the views respectively.\n\n## Pipeline\n\nThe API is centered around the **pipeline**, which most importantly consists\nout of the SQL program, but also has accompanying metadata and configuration parameters\n(e.g., compilation profile, number of workers, etc.).\n\n* A pipeline is identified and referred to by its user-provided unique name.\n* The pipeline program is asynchronously compiled when the pipeline is first created or\n when its program is subsequently updated.\n* Pipeline deployment start is only able to proceed to provisioning once the program is successfully\n compiled.\n* A pipeline cannot be updated while it is deployed.\n\n## Concurrency\n\nEach pipeline has a version, which is incremented each time its core fields are updated.\nThe version is monotonically increasing. There is additionally a program version which covers\nonly the program-related core fields, and is used by the compiler to discern when to recompile.\n\n## Client request handling\n\n### Request outcome expectations\n\nThe outcome of a request is that it either fails (e.g., DNS lookup failed) without any response\n(no status code nor body), or it succeeds and gets back a response status code and body.\n\nIn case of a response, usually it is the Feldera endpoint that generated it:\n- If it is success (2xx), it will return whichever body belongs to the success response.\n- Otherwise, if it is an error (4xx, 5xx), it will return a Feldera error response JSON body\n which will have an application-level `error_code`.\n\nHowever, there are two notable exceptions when the response is not generated by the Feldera\nendpoint:\n- If the HTTP server, to which the endpoint belongs, encountered an issue, it might return\n 4xx (e.g., for an unknown endpoint) or 5xx error codes by itself (e.g., when it is initializing).\n- If the Feldera API server is behind a (reverse) proxy, the proxy can return error codes by itself,\n for example BAD GATEWAY (502) or GATEWAY TIMEOUT (504).\n\nAs such, it is not guaranteed that the (4xx, 5xx) will have a Feldera error response JSON body\nin these latter cases.\n\n### Error handling and retrying\n\nThe error type returned by the client should distinguish between the error responses generated\nby Feldera endpoints themselves (which have a Feldera error response body) and those that are\ngenerated by other sources.\n\nIn order for a client operation (e.g., `pipeline.resume()`) to be robust (i.e., not fail due to\na single HTTP request not succeeding) the client should use a retry mechanism if the operation\nis idempotent. The retry mechanism must however have a time limit, after which it times out.\nThis guarantees that the client operation is eventually responsive, which enables the script\nit is a part of to not hang indefinitely on Feldera operations and instead be able to decide\nby itself whether and how to proceed. If no response is returned, the mechanism should generally\nretry. When a response is returned, the decision whether to retry can generally depend on the status\ncode: especially the status codes 408, 502, 503 and 504 should be considered as transient errors.\nFiner grained retry decisions should be made by taking into account the application-level\n`error_code` if the response body was indeed a Feldera error response body.\n\n## Feldera client errors (4xx)\n\n_Client behavior:_ clients should generally return with an error when they get back a 4xx status\ncode, as it usually means the request will likely not succeed even if it is sent again. Certain\nrequests might make use of a timed retry mechanism when the client error is transient without\nrequiring any user intervention to overcome, for instance a transaction already being in progress\nleading to a temporary CONFLICT (409) error.\n\n- **BAD REQUEST (400)**: invalid user request (general).\n - _Example:_ the new pipeline name `example1@~` contains invalid characters.\n\n- **UNAUTHORIZED (401)**: the user is not authorized to issue the request.\n - _Example:_ an invalid API key is provided.\n\n- **NOT FOUND (404)**: a resource required to exist in order to process the request was not found.\n - _Example:_ a pipeline named `example` does not exist when trying to update it.\n\n- **CONFLICT (409)**: there is a conflict between the request and a relevant resource.\n - _Example:_ a pipeline named `example` already exists.\n - _Example:_ another transaction is already in process.\n\n## Feldera server errors (5xx)\n\n- **INTERNAL SERVER ERROR (500)**: the server is unexpectedly unable to process the request\n (general).\n - _Example:_ unable to reach the database.\n - _Client behavior:_ immediately return with an error.\n\n- **NOT IMPLEMENTED (501)**: the server does not implement functionality required to process the\n request.\n - _Example:_ making a request to an enterprise-only endpoint in the OSS edition.\n - _Client behavior:_ immediately return with an error.\n\n- **SERVICE UNAVAILABLE (503)**: the server is not (yet) able to process the request.\n - _Example:_ pausing a pipeline which is still provisioning.\n - _Client behavior:_ depending on the type of request, client may use a timed retry mechanism.\n\n## Feldera error response body\n\nWhen the Feldera API returns an HTTP error status code (4xx, 5xx), the body will contain the\nfollowing JSON object:\n\n```json\n{\n \"message\": \"Human-readable explanation.\",\n \"error_code\": \"CodeSpecifyingError\",\n \"details\": {\n\n }\n}\n```\n\nIt contains the following fields:\n- **message (string)**: human-readable explanation of the error that occurred and potentially\n hinting what can be done about it.\n- **error_code (string)**: application-level code about the error that occurred, written in CamelCase.\n For example: `UnknownPipelineName`, `DuplicateName`, `PauseWhileNotProvisioned`, ... .\n- **details (object)**: JSON object corresponding to the `error_code` with fields that provide\n details relevant to it. For example: if a name is unknown, a field with the unknown name in\n question.\n"
contact:
name: Feldera Team
email: dev@feldera.com
license:
name: MIT OR Apache-2.0
version: 0.323.0
tags:
- name: Input Connectors
paths:
/v0/pipelines/{pipeline_name}/completion_status:
get:
tags:
- Input Connectors
summary: Check Completion Status
description: 'Check the status of a completion token returned by the `/ingress` or `/completion_token`
endpoint.'
operationId: completion_status
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: token
in: query
description: Completion token returned by the '/ingress' or '/completion_status' endpoint.
required: true
schema:
type: string
responses:
'200':
description: The pipeline has finished processing inputs associated with the provided completion token.
content:
application/json:
schema:
$ref: '#/components/schemas/CompletionStatusResponse'
'202':
description: The pipeline is still processing inputs associated with the provided completion token.
content:
application/json:
schema:
$ref: '#/components/schemas/CompletionStatusResponse'
'400':
description: An invalid completion token was provided
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'404':
description: Pipeline with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'410':
description: Completion token was created by a previous incarnation of the pipeline and is not valid for the current incarnation. This indicates that the pipeline was suspended and resumed from a checkpoint or restarted after a failure.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/ingress/{table_name}:
post:
tags:
- Input Connectors
summary: Insert Data
description: 'Push data to a SQL table.
The client sends data encoded using the format specified in the `?format=`
parameter as a body of the request. The contents of the data must match
the SQL table schema specified in `table_name`
The pipeline ingests data as it arrives without waiting for the end of
the request. Successful HTTP response indicates that all data has been
ingested successfully.
On success, returns a completion token that can be passed to the
''/completion_status'' endpoint to check whether the pipeline has fully
processed the data.'
operationId: http_input
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: table_name
in: path
description: SQL table name. Unquoted SQL names have to be capitalized. Quoted SQL names have to exactly match the case from the SQL program.
required: true
schema:
type: string
- name: force
in: query
description: When `true`, push data to the pipeline even if the pipeline is paused. The default value is `false`
required: true
schema:
type: boolean
- name: format
in: query
description: Input data format, either `csv' or 'json'.
required: true
schema:
type: string
- name: array
in: query
description: Set to `true` if updates in this stream are packaged into JSON arrays (used in conjunction with `format=json`). The default values is `false`.
required: false
schema:
type: boolean
nullable: true
- name: update_format
in: query
description: JSON data change event format (used in conjunction with `format=json`). The default value is 'insert_delete'.
required: false
schema:
allOf:
- $ref: '#/components/schemas/JsonUpdateFormat'
nullable: true
requestBody:
description: Input data in the specified format
content:
text/plain:
schema:
type: string
required: true
responses:
'200':
description: Data successfully delivered to the pipeline. The body of the response contains a completion token that can be passed to the '/completion_status' endpoint to check whether the pipeline has fully processed the data.
content:
application/json:
schema:
$ref: '#/components/schemas/CompletionTokenResponse'
'400':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'404':
description: Pipeline and/or table with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Pipeline with that name does not exist:
value:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/completion_token:
get:
tags:
- Input Connectors
summary: Get Completion Token
description: 'Generate a completion token for an input connector.
Returns a token that can be passed to the `/completion_status` endpoint
to check whether the pipeline has finished processing all inputs received from the
connector before the token was generated.'
operationId: completion_token
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: table_name
in: path
description: SQL table name. Unquoted SQL names have to be capitalized. Quoted SQL names have to exactly match the case from the SQL program.
required: true
schema:
type: string
- name: connector_name
in: path
description: Unique input connector name
required: true
schema:
type: string
responses:
'200':
description: Completion token that can be passed to the '/completion_status' endpoint.
content:
application/json:
schema:
$ref: '#/components/schemas/CompletionTokenResponse'
'404':
description: Specified pipeline, table, or connector does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/stats:
get:
tags:
- Input Connectors
summary: Get Input Status
description: Retrieve the status of an input connector.
operationId: get_pipeline_input_connector_status
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: table_name
in: path
description: Unique table name
required: true
schema:
type: string
- name: connector_name
in: path
description: Unique input connector name
required: true
schema:
type: string
responses:
'200':
description: Input connector status retrieved successfully
content:
application/json:
schema:
$ref: '#/components/schemas/InputEndpointStatus'
'404':
description: Pipeline, table and/or input connector with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Pipeline with that name does not exist:
value:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
details:
pipeline_name: non-existent-pipeline
'500':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'503':
description: ''
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Disconnected during response:
value:
message: 'Error sending HTTP request to pipeline: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs. Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: the pipeline disconnected while it was processing this HTTP request. This could be because the pipeline either (a) encountered a fatal error or panic, (b) was stopped, or (c) experienced network issues -- retrying might help in the last case. Alternatively, check the pipeline logs.
Pipeline is currently unavailable:
value:
message: 'Error sending HTTP request to pipeline: deployment status is currently ''unavailable'' -- wait for it to become ''running'' or ''paused'' again Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: deployment status is currently 'unavailable' -- wait for it to become 'running' or 'paused' again
Pipeline is not deployed:
value:
message: Unable to interact with pipeline because the deployment status (stopped) indicates it is not (yet) fully provisioned pipeline-id=N/A pipeline-name="my_pipeline"
error_code: PipelineInteractionNotDeployed
details:
pipeline_name: my_pipeline
status: Stopped
desired_status: Provisioned
Response timeout:
value:
message: 'Error sending HTTP request to pipeline: timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response) Failed request: /pause pipeline-id=N/A pipeline-name="my_pipeline"'
error_code: PipelineInteractionUnreachable
details:
pipeline_name: my_pipeline
request: /pause
error: 'timeout (10s) was reached: this means the pipeline took too long to respond -- this can simply be because the request was too difficult to process in time, or other reasons (e.g., deadlock): the pipeline logs might contain additional information (original send request error: Timeout while waiting for response)'
security:
- JSON web token (JWT) or API key: []
/v0/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/{action}:
post:
tags:
- Input Connectors
summary: Control Input Connector
description: 'Start (resume) or pause the input connector.
The following values of the `action` argument are accepted: `start` and `pause`.
Input connectors can be in either the `Running` or `Paused` state. By default,
connectors are initialized in the `Running` state when a pipeline is deployed.
In this state, the connector actively fetches data from its configured data
source and forwards it to the pipeline. If needed, a connector can be created
in the `Paused` state by setting its
[`paused`](https://docs.feldera.com/connectors/#generic-attributes) property
to `true`. When paused, the connector remains idle until reactivated using the
`start` command. Conversely, a connector in the `Running` state can be paused
at any time by issuing the `pause` command.
The current connector state can be retrieved via the
`GET /v0/pipelines/{pipeline_name}/stats` endpoint.
Note that only if both the pipeline *and* the connector state is `Running`,
is the input connector active.
```text
Pipeline state Connector state Connector is active?
-------------- --------------- --------------------
Paused Paused No
Paused Running No
Running Paused No
Running Running Yes
```'
operationId: post_pipeline_input_connector_action
parameters:
- name: pipeline_name
in: path
description: Unique pipeline name
required: true
schema:
type: string
- name: table_name
in: path
description: SQL table name
required: true
schema:
type: string
- name: connector_name
in: path
description: Input connector name
required: true
schema:
type: string
- name: action
in: path
required: true
schema:
type: string
responses:
'200':
description: Action has been processed
'404':
description: Pipeline, table and/or input connector with that name does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
examples:
Pipeline with that name does not exist:
value:
message: Unknown pipeline name 'non-existent-pipeline'
error_code: UnknownPipelineName
detail
# --- truncated at 32 KB (45 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/feldera/refs/heads/main/openapi/feldera-input-connectors-api-openapi.yml