The Dataset API from Apache Airflow — 4 operation(s) for dataset.
Every API here is available over the APIs.io API and to AI agents over MCP.
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.curl "https://apis.io/api/v1/apis/apache-airflow:apache-airflow-dataset-api"curl "https://apis.io/api/v1/apis?limit=25"Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.
Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.
A second provider on the same verified email joins the account you already have.
openapi: 3.2.0
info:
contact:
email: dev@airflow.apache.org
name: Apache Software Foundation
url: https://airflow.apache.org
x-twitter: TheASF
description: '# Overview
To facilitate management, Apache Airflow supports a range of REST API endpoints across its
objects. This section provides an overview of the API design, methods, and supported use cases. Most of the endpoints accept `JSON` as input and return `JSON` responses. This means that you must usually add the following headers to your request:
```
Content-type: application/json
Accept: application/json
```
## Resources
The term `resource` refers to a single type of object in the Airflow metadata. An API is broken up by its
endpoint''s corresponding resource. The name of a resource is typically plural and expressed in camelCase. Example: `dagRuns`. Resource names are used as part of endpoint URLs, as well as in API parameters and responses. ## CRUD Operations
The platform supports **C**reate, **R**ead, **U**pdate, and **D**elete operations on most resources. You can review the standards for these operations and their standard parameters below. Some endpoints have special behavior as exceptions. ### Create
To create a resource, you typically submit an HTTP `POST` request with the resource''s required metadata
in the request body. The response returns a `201 Created` response code upon success with the resource''s metadata, including
its internal `id`, in the response body. ### Read
The HTTP `GET` request can be used to read a resource or to list a number of resources. A resource''s `id` can be submitted in the request parameters to read a specific resource. The response usually returns a `200 OK` response code upon success, with the resource''s metadata in
the response body. If a `GET` request does not include a specific resource `id`, it is treated as a list request. The response usually returns a `200 OK` response code upon success, with an object containing a list
of resources'' metadata in the response body. When reading resources, some common query parameters are usually available. e.g.:
```
v1/connections?limit=25&offset=25
```
|Query Parameter|Type|Description|
|---------------|----|-----------|
|limit|integer|Maximum number of objects to fetch. Usually 25 by default|
|offset|integer|Offset after which to start returning objects. For use with limit query parameter.|
### Update
Updating a resource requires the resource `id`, and is typically done using an HTTP `PATCH` request,
with the fields to modify in the request body. The response usually returns a `200 OK` response code upon success, with information about the modified
resource in the response body. ### Delete
Deleting a resource requires the resource `id` and is typically executing via an HTTP `DELETE` request. The response usually returns a `204 No Content` response code upon success. ## Conventions
- Resource names are plural and expressed in camelCase. - Names are consistent between URL parameter name and field name. - Field names are in snake_case. ```json
{
"name": "string",
"slots": 0,
"occupied_slots": 0,
"used_slots": 0,
"queued_slots": 0,
"open_slots": 0
}
```
### Update Mask
Update mask is available as a query parameter in patch endpoints. It is used to notify the
API which fields you want to update. Using `update_mask` makes it easier to update objects
by helping the server know which fields to update in an object instead of updating all fields. The update request ignores any fields that aren''t specified in the field mask, leaving them with
their current values. Example:
```
resource = request.get(''/resource/my-id'').json()
resource[''my_field''] = ''new-value''
request.patch(''/resource/my-id?update_mask=my_field'', data=json.dumps(resource))
```
## Versioning and Endpoint Lifecycle
- API versioning is not synchronized to specific releases of the Apache Airflow. - APIs are designed to be backward compatible. - Any changes to the API will first go through a deprecation phase. # Trying the API
You can use a third party client, such as curl, HTTPie,
Postman or the Insomnia rest client to test
the Apache Airflow API.'
license:
name: Apache 2.0
url: http://www.apache.org/licenses/LICENSE-2.0.html
title: Airflow API (Stable) Dataset API
version: 2.5.3
x-apisguru-categories:
- messaging
x-logo:
url: https://twitter.com/TheASF/profile_image?size=original
x-origin:
- format: openapi
url: https://airflow.apache.org/docs/apache-airflow/stable/_specs/v1.yaml
version: '3.0'
x-providerName: apache.org
x-serviceName: airflow
servers:
- description: Apache Airflow Stable API.
url: /api/v1
- url: https://apache.local
security: []
tags:
- name: Dataset
paths:
/dags/{dag_id}/dagRuns/{dag_run_id}/upstreamDatasetEvents:
parameters:
- $ref: '#/components/parameters/DAGID'
- $ref: '#/components/parameters/DAGRunID'
get:
description: 'Get datasets for a dag run.
*New in version 2.4.0*'
operationId: get_upstream_dataset_events
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetEventCollection'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
'404':
$ref: '#/components/responses/NotFound'
summary: Apache Airflow Get Dataset Events for a DAG Run
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dag_run_endpoint
/datasets:
get:
operationId: get_datasets
parameters:
- $ref: '#/components/parameters/PageLimit'
- $ref: '#/components/parameters/PageOffset'
- $ref: '#/components/parameters/OrderBy'
- description: 'If set, only return datasets with uris matching this pattern.
'
in: query
name: uri_pattern
required: false
schema:
type: string
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetCollection'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
summary: Apache Airflow List Datasets
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dataset_endpoint
/datasets/events:
parameters:
- $ref: '#/components/parameters/PageLimit'
- $ref: '#/components/parameters/PageOffset'
- $ref: '#/components/parameters/OrderBy'
- $ref: '#/components/parameters/FilterDatasetID'
- $ref: '#/components/parameters/FilterSourceDAGID'
- $ref: '#/components/parameters/FilterSourceTaskID'
- $ref: '#/components/parameters/FilterSourceRunID'
- $ref: '#/components/parameters/FilterSourceMapIndex'
get:
description: Get dataset events
operationId: get_dataset_events
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetEventCollection'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
'404':
$ref: '#/components/responses/NotFound'
summary: Apache Airflow Get Dataset Events
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dataset_endpoint
/datasets/{uri}:
parameters:
- $ref: '#/components/parameters/DatasetURI'
get:
description: Get a dataset by uri.
operationId: get_dataset
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Dataset'
description: Success.
'401':
$ref: '#/components/responses/Unauthenticated'
'403':
$ref: '#/components/responses/PermissionDenied'
'404':
$ref: '#/components/responses/NotFound'
summary: Apache Airflow Get a Dataset
tags:
- Dataset
x-openapi-router-controller: airflow.api_connexion.endpoints.dataset_endpoint
components:
parameters:
FilterDatasetID:
description: The Dataset ID that updated the dataset.
in: query
name: dataset_id
schema:
type: integer
FilterSourceMapIndex:
description: The map index that updated the dataset.
in: query
name: source_map_index
schema:
type: integer
DAGID:
description: The DAG ID.
in: path
name: dag_id
required: true
schema:
type: string
DAGRunID:
description: The DAG run ID.
in: path
name: dag_run_id
required: true
schema:
type: string
DatasetURI:
description: The encoded Dataset URI
in: path
name: uri
required: true
schema:
format: path
type: string
PageLimit:
description: The numbers of items to return.
in: query
name: limit
required: false
schema:
default: 100
type: integer
PageOffset:
description: The number of items to skip before starting to collect the result set.
in: query
name: offset
required: false
schema:
minimum: 0
type: integer
OrderBy:
description: 'The name of the field to order the results by.
Prefix a field name with `-` to reverse the sort order.
*New in version 2.1.0*
'
in: query
name: order_by
required: false
schema:
type: string
FilterSourceDAGID:
description: The DAG ID that updated the dataset.
in: query
name: source_dag_id
schema:
type: string
FilterSourceRunID:
description: The DAG run ID that updated the dataset.
in: query
name: source_run_id
schema:
type: string
FilterSourceTaskID:
description: The task ID that updated the dataset.
in: query
name: source_task_id
schema:
type: string
responses:
NotFound:
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
description: A specified resource is not found.
Unauthenticated:
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
description: Request not authenticated due to missing, invalid, authentication info.
PermissionDenied:
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
description: Client does not have sufficient permission.
schemas:
BasicDAGRun:
properties:
dag_id:
readOnly: true
type: string
data_interval_end:
format: date-time
readOnly: true
type:
- string
- 'null'
data_interval_start:
format: date-time
readOnly: true
type:
- string
- 'null'
end_date:
format: date-time
readOnly: true
type:
- string
- 'null'
logical_date:
description: 'The logical date (previously called execution date). This is the time or interval covered by
this DAG run, according to the DAG definition.
The value of this field can be set only when creating the object. If you try to modify the
field of an existing object, the request fails with an BAD_REQUEST error.
This together with DAG_ID are a unique key.
*New in version 2.2.0*
'
format: date-time
type: string
run_id:
description: 'Run ID.
'
type: string
start_date:
description: 'The start time. The time when DAG run was actually created.
*Changed in version 2.1.3*: Field becomes nullable.
'
format: date-time
readOnly: true
type:
- string
- 'null'
state:
$ref: '#/components/schemas/DagState'
type: object
CollectionInfo:
description: Metadata about collection.
properties:
total_entries:
description: 'Count of total objects in the current result set before pagination parameters
(limit, offset) are applied.
'
type: integer
type: object
Dataset:
description: 'A dataset item.
*New in version 2.4.0*
'
properties:
consuming_dags:
items:
$ref: '#/components/schemas/DagScheduleDatasetReference'
type: array
created_at:
description: The dataset creation time
type: string
extra:
description: The dataset extra
type:
- object
- 'null'
id:
description: The dataset id
type: integer
producing_tasks:
items:
$ref: '#/components/schemas/TaskOutletDatasetReference'
type: array
updated_at:
description: The dataset update time
type: string
uri:
description: The dataset uri
type: string
type: object
DagScheduleDatasetReference:
description: 'A datasets reference to a downstream DAG.
*New in version 2.4.0*
'
properties:
created_at:
description: The dataset reference creation time
type: string
dag_id:
description: The DAG ID that depends on the dataset.
type:
- string
- 'null'
updated_at:
description: The dataset reference update time
type: string
type: object
DatasetEventCollection:
allOf:
- properties:
dataset_events:
items:
$ref: '#/components/schemas/DatasetEvent'
type: array
type: object
- $ref: '#/components/schemas/CollectionInfo'
description: 'A collection of dataset events.
*New in version 2.4.0*
'
type: object
Error:
description: '[RFC7807](https://tools.ietf.org/html/rfc7807) compliant response.
'
properties:
detail:
description: A human-readable explanation specific to this occurrence of the problem.
type: string
instance:
description: 'A URI reference that identifies the specific occurrence of the problem. It may or may
not yield further information if dereferenced.
'
type: string
status:
description: The HTTP status code generated by the API server for this occurrence of the problem.
type: number
title:
description: A short, human-readable summary of the problem type.
type: string
type:
description: 'A URI reference [RFC3986] that identifies the problem type. This specification
encourages that, when dereferenced, it provide human-readable documentation for
the problem type.
'
type: string
required:
- type
- title
- status
type: object
TaskOutletDatasetReference:
description: 'A datasets reference to an upstream task.
*New in version 2.4.0*
'
properties:
created_at:
description: The dataset creation time
type: string
dag_id:
description: The DAG ID that updates the dataset.
type:
- string
- 'null'
task_id:
description: The task ID that updates the dataset.
type:
- string
- 'null'
updated_at:
description: The dataset update time
type: string
type: object
DatasetCollection:
allOf:
- properties:
datasets:
items:
$ref: '#/components/schemas/Dataset'
type: array
type: object
- $ref: '#/components/schemas/CollectionInfo'
description: 'A collection of datasets.
*New in version 2.4.0*
'
type: object
DatasetEvent:
description: 'A dataset event.
*New in version 2.4.0*
'
properties:
created_dagruns:
items:
$ref: '#/components/schemas/BasicDAGRun'
type: array
dataset_id:
description: The dataset id
type: integer
dataset_uri:
description: The URI of the dataset
type: string
extra:
description: The dataset event extra
type:
- object
- 'null'
source_dag_id:
description: The DAG ID that updated the dataset.
type:
- string
- 'null'
source_map_index:
description: The task map index that updated the dataset.
type:
- integer
- 'null'
source_run_id:
description: The DAG run ID that updated the dataset.
type:
- string
- 'null'
source_task_id:
description: The task ID that updated the dataset.
type:
- string
- 'null'
timestamp:
description: The dataset event creation time
type: string
type: object
DagState:
description: 'DAG State.
*Changed in version 2.1.3*: ''queued'' is added as a possible value.
'
enum:
- queued
- running
- success
- failed
readOnly: true
type: string
securitySchemes:
Basic:
scheme: basic
type: http
GoogleOpenId:
openIdConnectUrl: https://accounts.google.com/.well-known/openid-configuration
type: openIdConnect
Kerberos:
scheme: negotiate
type: http
externalDocs:
url: https://airflow.apache.org/docs/apache-airflow/stable/