The Dataset API from Apache Airflow — 4 operation(s) for dataset.
openapi: 3.0.3
info:
contact:
email: dev@airflow.apache.org
name: Apache Software Foundation
url: https://airflow.apache.org
x-twitter: TheASF
description: "# Overview\n\nTo facilitate management, Apache Airflow supports a range of REST API endpoints across its\nobjects.\nThis section provides an overview of the API design, methods, and supported use cases.\n\nMost of the endpoints accept `JSON` as input and return `JSON` responses.\nThis means that you must usually add the following headers to your request:\n```\nContent-type: application/json\nAccept: application/json\n```\n\n## Resources\n\nThe term `resource` refers to a single type of object in the Airflow metadata. An API is broken up by its\nendpoint's corresponding resource.\nThe name of a resource is typically plural and expressed in camelCase. Example: `dagRuns`.\n\nResource names are used as part of endpoint URLs, as well as in API parameters and responses.\n\n## CRUD Operations\n\nThe platform supports **C**reate, **R**ead, **U**pdate, and **D**elete operations on most resources.\nYou can review the standards for these operations and their standard parameters below.\n\nSome endpoints have special behavior as exceptions.\n\n### Create\n\nTo create a resource, you typically submit an HTTP `POST` request with the resource's required metadata\nin the request body.\nThe response returns a `201 Created` response code upon success with the resource's metadata, including\nits internal `id`, in the response body.\n\n### Read\n\nThe HTTP `GET` request can be used to read a resource or to list a number of resources.\n\nA resource's `id` can be submitted in the request parameters to read a specific resource.\nThe response usually returns a `200 OK` response code upon success, with the resource's metadata in\nthe response body.\n\nIf a `GET` request does not include a specific resource `id`, it is treated as a list request.\nThe response usually returns a `200 OK` response code upon success, with an object containing a list\nof resources' metadata in the response body.\n\nWhen reading resources, some common query parameters are usually available. e.g.:\n```\nv1/connections?limit=25&offset=25\n```\n\n|Query Parameter|Type|Description|\n|---------------|----|-----------|\n|limit|integer|Maximum number of objects to fetch. Usually 25 by default|\n|offset|integer|Offset after which to start returning objects. For use with limit query parameter.|\n\n### Update\n\nUpdating a resource requires the resource `id`, and is typically done using an HTTP `PATCH` request,\nwith the fields to modify in the request body.\nThe response usually returns a `200 OK` response code upon success, with information about the modified\nresource in the response body.\n\n### Delete\n\nDeleting a resource requires the resource `id` and is typically executing via an HTTP `DELETE` request.\nThe response usually returns a `204 No Content` response code upon success.\n\n## Conventions\n\n- Resource names are plural and expressed in camelCase.\n- Names are consistent between URL parameter name and field name.\n\n- Field names are in snake_case.\n```json\n{\n \"name\": \"string\",\n \"slots\": 0,\n \"occupied_slots\": 0,\n \"used_slots\": 0,\n \"queued_slots\": 0,\n \"open_slots\": 0\n}\n```\n\n### Update Mask\n\nUpdate mask is available as a query parameter in patch endpoints. It is used to notify the\nAPI which fields you want to update. Using `update_mask` makes it easier to update objects\nby helping the server know which fields to update in an object instead of updating all fields.\nThe update request ignores any fields that aren't specified in the field mask, leaving them with\ntheir current values.\n\nExample:\n```\n resource = request.get('/resource/my-id').json()\n resource['my_field'] = 'new-value'\n request.patch('/resource/my-id?update_mask=my_field', data=json.dumps(resource))\n```\n\n## Versioning and Endpoint Lifecycle\n\n- API versioning is not synchronized to specific releases of the Apache Airflow.\n- APIs are designed to be backward compatible.\n- Any changes to the API will first go through a deprecation phase.\n\n# Trying the API\n\nYou can use a third party client, such as [curl](https://curl.haxx.se/), [HTTPie](https://httpie.org/),\n[Postman](https://www.postman.com/) or [the Insomnia rest client](https://insomnia.rest/) to test\nthe Apache Airflow API.\n\nNote that you will need to pass credentials data.\n\nFor e.g., here is how to pause a DAG with [curl](https://curl.haxx.se/), when basic authorization is used:\n```bash\ncurl -X PATCH 'https://example.com/api/v1/dags/{dag_id}?update_mask=is_paused' \\\n-H 'Content-Type: application/json' \\\n--user \"username:password\" \\\n-d '{\n \"is_paused\": true\n}'\n```\n\nUsing a graphical tool such as [Postman](https://www.postman.com/) or [Insomnia](https://insomnia.rest/),\nit is possible to import the API specifications directly:\n\n1. Download the API specification by clicking the **Download** button at top of this document\n2. Import the JSON specification in the graphical tool of your choice.\n - In *Postman*, you can click the **import** button at the top\n - With *Insomnia*, you can just drag-and-drop the file on the UI\n\nNote that with *Postman*, you can also generate code snippets by selecting a request and clicking on\nthe **Code** button.\n\n## Enabling CORS\n\n[Cross-origin resource sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)\nis a browser security feature that restricts HTTP requests that are\ninitiated from scripts running in the browser.\n\nFor details on enabling/configuring CORS, see\n[Enabling CORS](https://airflow.apache.org/docs/apache-airflow/stable/security/api.html).\n\n# Authentication\n\nTo be able to meet the requirements of many organizations, Airflow supports many authentication methods,\nand it is even possible to add your own method.\n\nIf you want to check which auth backend is currently set, you can use\n`airflow config get-value api auth_backends` command as in the example below.\n```bash\n$ airflow config get-value api auth_backends\nairflow.api.auth.backend.basic_auth\n```\nThe default is to deny all requests.\n\nFor details on configuring the authentication, see\n[API Authorization](https://airflow.apache.org/docs/apache-airflow/stable/security/api.html).\n\n# Errors\n\nWe follow the error response format proposed in [RFC 7807](https://tools.ietf.org/html/rfc7807)\nalso known as Problem Details for HTTP APIs. As with our normal API responses,\nyour client must be prepared to gracefully handle additional members of the response.\n\n## Unauthenticated\n\nThis indicates that the request has not been applied because it lacks valid authentication\ncredentials for the target resource. Please check that you have valid credentials.\n\n## PermissionDenied\n\nThis response means that the server understood the request but refuses to authorize\nit because it lacks sufficient rights to the resource. It happens when you do not have the\nnecessary permission to execute the action you performed. You need to get the appropriate\npermissions in other to resolve this error.\n\n## BadRequest\n\nThis response means that the server cannot or will not process the request due to something\nthat is perceived to be a client error (e.g., malformed request syntax, invalid request message\nframing, or deceptive request routing). To resolve this, please ensure that your syntax is correct.\n\n## NotFound\n\nThis client error response indicates that the server cannot find the requested resource.\n\n## MethodNotAllowed\n\nIndicates that the request method is known by the server but is not supported by the target resource.\n\n## NotAcceptable\n\nThe target resource does not have a current representation that would be acceptable to the user\nagent, according to the proactive negotiation header fields received in the request, and the\nserver is unwilling to supply a default representation.\n\n## AlreadyExists\n\nThe request could not be completed due to a conflict with the current state of the target\nresource, e.g. the resource it tries to create already exists.\n\n## Unknown\n\nThis means that the server encountered an unexpected condition that prevented it from\nfulfilling the request.\n"
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
title: Airflow API (Stable) Config Dataset API
version: 2.5.3
x-apisguru-categories:
- messaging
x-logo:
url: https://twitter.com/TheASF/profile_image?size=original
x-origin:
- format: openapi
url: https://airflow.apache.org/docs/apache-airflow/stable/_specs/v1.yaml
version: '3.0'
x-providerName: apache.org
x-serviceName: airflow
servers:
- description: Apache Airflow Stable API.
url: /api/v1
- url: https://apache.local
security: []
tags:
- name: Dataset
paths:
/dags/{dag_id}/dagRuns/{dag_run_id}/upstreamDatasetEvents:
parameters:
- $ref: '#/components/parameters/DAGID'
- $ref: '#/components/parameters/DAGRunID'
get:
description: 'Get datasets for a dag run.
*New in version 2.4.0*
'
operationId: get_upstream_dataset_events
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetEventCollection'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
'404':
$ref: '#/components/responses/NotFound'
summary: Apache Airflow Get Dataset Events for a DAG Run
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dag_run_endpoint
/datasets:
get:
operationId: get_datasets
parameters:
- $ref: '#/components/parameters/PageLimit'
- $ref: '#/components/parameters/PageOffset'
- $ref: '#/components/parameters/OrderBy'
- description: 'If set, only return datasets with uris matching this pattern.
'
in: query
name: uri_pattern
required: false
schema:
type: string
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetCollection'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
summary: Apache Airflow List Datasets
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dataset_endpoint
/datasets/events:
parameters:
- $ref: '#/components/parameters/PageLimit'
- $ref: '#/components/parameters/PageOffset'
- $ref: '#/components/parameters/OrderBy'
- $ref: '#/components/parameters/FilterDatasetID'
- $ref: '#/components/parameters/FilterSourceDAGID'
- $ref: '#/components/parameters/FilterSourceTaskID'
- $ref: '#/components/parameters/FilterSourceRunID'
- $ref: '#/components/parameters/FilterSourceMapIndex'
get:
description: Get dataset events
operationId: get_dataset_events
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetEventCollection'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
'404':
$ref: '#/components/responses/NotFound'
summary: Apache Airflow Get Dataset Events
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dataset_endpoint
/datasets/{uri}:
parameters:
- $ref: '#/components/parameters/DatasetURI'
get:
description: Get a dataset by uri.
operationId: get_dataset
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Dataset'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
'404':
$ref: '#/components/responses/NotFound'
summary: Apache Airflow Get a Dataset
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dataset_endpoint
components:
schemas:
TaskOutletDatasetReference:
description: 'A datasets reference to an upstream task.
*New in version 2.4.0*
'
properties:
created_at:
description: The dataset creation time
nullable: false
type: string
dag_id:
description: The DAG ID that updates the dataset.
nullable: true
type: string
task_id:
description: The task ID that updates the dataset.
nullable: true
type: string
updated_at:
description: The dataset update time
nullable: false
type: string
type: object
DatasetEventCollection:
allOf:
- properties:
dataset_events:
items:
$ref: '#/components/schemas/DatasetEvent'
type: array
type: object
- $ref: '#/components/schemas/CollectionInfo'
description: 'A collection of dataset events.
*New in version 2.4.0*
'
type: object
DagScheduleDatasetReference:
description: 'A datasets reference to a downstream DAG.
*New in version 2.4.0*
'
properties:
created_at:
description: The dataset reference creation time
nullable: false
type: string
dag_id:
description: The DAG ID that depends on the dataset.
nullable: true
type: string
updated_at:
description: The dataset reference update time
nullable: false
type: string
type: object
DagState:
description: 'DAG State.
*Changed in version 2.1.3*: ''queued'' is added as a possible value.
'
enum:
- queued
- running
- success
- failed
readOnly: true
type: string
CollectionInfo:
description: Metadata about collection.
properties:
total_entries:
description: 'Count of total objects in the current result set before pagination parameters
(limit, offset) are applied.
'
type: integer
type: object
Error:
description: '[RFC7807](https://tools.ietf.org/html/rfc7807) compliant response.
'
properties:
detail:
description: A human-readable explanation specific to this occurrence of the problem.
type: string
instance:
description: 'A URI reference that identifies the specific occurrence of the problem. It may or may
not yield further information if dereferenced.
'
type: string
status:
description: The HTTP status code generated by the API server for this occurrence of the problem.
type: number
title:
description: A short, human-readable summary of the problem type.
type: string
type:
description: 'A URI reference [RFC3986] that identifies the problem type. This specification
encourages that, when dereferenced, it provide human-readable documentation for
the problem type.
'
type: string
required:
- type
- title
- status
type: object
Dataset:
description: 'A dataset item.
*New in version 2.4.0*
'
properties:
consuming_dags:
items:
$ref: '#/components/schemas/DagScheduleDatasetReference'
type: array
created_at:
description: The dataset creation time
nullable: false
type: string
extra:
description: The dataset extra
nullable: true
type: object
id:
description: The dataset id
type: integer
producing_tasks:
items:
$ref: '#/components/schemas/TaskOutletDatasetReference'
type: array
updated_at:
description: The dataset update time
nullable: false
type: string
uri:
description: The dataset uri
nullable: false
type: string
type: object
DatasetEvent:
description: 'A dataset event.
*New in version 2.4.0*
'
properties:
created_dagruns:
items:
$ref: '#/components/schemas/BasicDAGRun'
type: array
dataset_id:
description: The dataset id
type: integer
dataset_uri:
description: The URI of the dataset
nullable: false
type: string
extra:
description: The dataset event extra
nullable: true
type: object
source_dag_id:
description: The DAG ID that updated the dataset.
nullable: true
type: string
source_map_index:
description: The task map index that updated the dataset.
nullable: true
type: integer
source_run_id:
description: The DAG run ID that updated the dataset.
nullable: true
type: string
source_task_id:
description: The task ID that updated the dataset.
nullable: true
type: string
timestamp:
description: The dataset event creation time
nullable: false
type: string
type: object
BasicDAGRun:
properties:
dag_id:
readOnly: true
type: string
data_interval_end:
format: date-time
nullable: true
readOnly: true
type: string
data_interval_start:
format: date-time
nullable: true
readOnly: true
type: string
end_date:
format: date-time
nullable: true
readOnly: true
type: string
logical_date:
description: 'The logical date (previously called execution date). This is the time or interval covered by
this DAG run, according to the DAG definition.
The value of this field can be set only when creating the object. If you try to modify the
field of an existing object, the request fails with an BAD_REQUEST error.
This together with DAG_ID are a unique key.
*New in version 2.2.0*
'
format: date-time
type: string
run_id:
description: 'Run ID.
'
type: string
start_date:
description: 'The start time. The time when DAG run was actually created.
*Changed in version 2.1.3*: Field becomes nullable.
'
format: date-time
nullable: true
readOnly: true
type: string
state:
$ref: '#/components/schemas/DagState'
type: object
DatasetCollection:
allOf:
- properties:
datasets:
items:
$ref: '#/components/schemas/Dataset'
type: array
type: object
- $ref: '#/components/schemas/CollectionInfo'
description: 'A collection of datasets.
*New in version 2.4.0*
'
type: object
responses:
Unauthenticated:
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
description: Request not authenticated due to missing, invalid, authentication info.
PermissionDenied:
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
description: Client does not have sufficient permission.
NotFound:
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
description: A specified resource is not found.
parameters:
FilterSourceRunID:
description: The DAG run ID that updated the dataset.
in: query
name: source_run_id
schema:
type: string
FilterDatasetID:
description: The Dataset ID that updated the dataset.
in: query
name: dataset_id
schema:
type: integer
DAGRunID:
description: The DAG run ID.
in: path
name: dag_run_id
required: true
schema:
type: string
OrderBy:
description: 'The name of the field to order the results by.
Prefix a field name with `-` to reverse the sort order.
*New in version 2.1.0*
'
in: query
name: order_by
required: false
schema:
type: string
FilterSourceMapIndex:
description: The map index that updated the dataset.
in: query
name: source_map_index
schema:
type: integer
PageLimit:
description: The numbers of items to return.
in: query
name: limit
required: false
schema:
default: 100
type: integer
FilterSourceDAGID:
description: The DAG ID that updated the dataset.
in: query
name: source_dag_id
schema:
type: string
FilterSourceTaskID:
description: The task ID that updated the dataset.
in: query
name: source_task_id
schema:
type: string
PageOffset:
description: The number of items to skip before starting to collect the result set.
in: query
name: offset
required: false
schema:
minimum: 0
type: integer
DAGID:
description: The DAG ID.
in: path
name: dag_id
required: true
schema:
type: string
DatasetURI:
description: The encoded Dataset URI
in: path
name: uri
required: true
schema:
format: path
type: string
securitySchemes:
Basic:
scheme: basic
type: http
GoogleOpenId:
openIdConnectUrl: https://accounts.google.com/.well-known/openid-configuration
type: openIdConnect
Kerberos:
scheme: negotiate
type: http
externalDocs:
url: https://airflow.apache.org/docs/apache-airflow/stable/