Represents the primary data abstraction within the data platform, serving as a structured collection of data points. Datasets can be either manually uploaded or generated through data transformations, providing a versatile means for organizing, storing, and retrieving data across various use cases.
openapi: 3.2.0
info:
title: PhariaData Datasets API
description: The PhariaData API provides a comprehensive suite of endpoints to manage data workflows within the Pharia Data Platform. This API enables users to organize, store, retrieve, and manipulate datasets across repositories with efficiency and control. Key functionalities include the creation and management of datasets, stages, and repositories, as well as streamlined access to data through downloads and real-time data streaming.
contact:
name: PhariaData API Support
url: https://aleph-alpha.atlassian.net/servicedesk/customer/portals
email: pharia-data@aleph-alpha.com
license:
name: Open Aleph License
version: 1.0.0
servers:
- url: '{host}/v1/studio/data'
variables:
host:
default: https://api.pharia.example.com
security:
- Bearer: []
tags:
- name: Datasets
description: Represents the primary data abstraction within the data platform, serving as a structured collection of data points. Datasets can be either manually uploaded or generated through data transformations, providing a versatile means for organizing, storing, and retrieving data across various use cases.
paths:
/repositories/{repositoryID}/datasets:
get:
tags:
- Datasets
summary: Retrieve a list of datasets
description: Lists all datasets within a specified repository, with options to filter by creation date and labels. Use this endpoint to browse datasets and locate relevant data collections by repository. Returns 404 if the repository does not exist.
parameters:
- name: repositoryID
in: path
description: The unique identifier of the repository containing the datasets.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
- name: created_after
in: query
description: Filters datasets created after the specified date (in ISO 8601 format).
schema:
type: string
example: '2024-01-01'
- name: created_before
in: query
description: Filters datasets created before the specified date (in ISO 8601 format).
schema:
type: string
example: '2024-12-31'
- name: label
in: query
description: Filters datasets by one or more labels. Use this to narrow down results to datasets tagged with specific labels.
style: form
explode: false
schema:
type: array
items:
type: string
example:
- pharia-finetuning
- name: page
in: query
description: The page number to retrieve, used for paginated responses.
schema:
type: integer
example: 0
- name: size
in: query
description: The number of items to include per page in the response.
schema:
type: integer
example: 10
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetListResponse'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'404':
description: Repository not found - The specified repository does not exist
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'500':
description: INTERNAL SERVER ERROR
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
post:
tags:
- Datasets
summary: Create a dataset
description: Adds a new dataset to the specified repository. This endpoint accepts a file containing dataset content in JSON Lines format, along with optional license, metadata, and labels to better classify and describe the dataset.
parameters:
- name: repositoryID
in: path
description: Unique identifier of the repository in which the dataset will be created.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
requestBody:
required: true
content:
multipart/form-data:
schema:
type: object
required:
- sourceData
- totalDatapoints
properties:
sourceData:
type: string
description: File containing dataset content, serialized in JSON Lines format. Each line represents a distinct data point in JSON format.
format: binary
name:
type: string
description: Name assigned to the dataset for identification.
example: pharia-llm-dataset-evaluation
metadata:
type: object
description: Additional key-value metadata related to the dataset, used to provide descriptive information and context.
example:
source: studio
task: evaluation-pharia-llm-1
license:
type: object
description: License information for the dataset, ensuring transparency and regulatory compliance. Recommended licenses include the Open Aleph License (OAL-1.0), Creative Commons CC0, and Open Data Commons PDDL. If included, please provide the license ID and URL.
example:
id: OAL-1.0
url: https://github.com/Aleph-Alpha/.github/blob/main/OAL-1.0.pdf
labels:
type: string
description: A comma-separated list of labels associated with the dataset to enhance searchability and classification.
example: finetuning, pharia, aleph-alpha
totalDatapoints:
type: integer
description: Total number of data points contained in the dataset, useful for tracking dataset size.
example: 1000
responses:
'201':
description: Created
content:
application/json:
schema:
$ref: '#/components/schemas/Dataset'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'404':
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'422':
description: UnprocessableEntity
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'500':
description: INTERNAL SERVER ERROR
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
/repositories/{repositoryID}/datasets/{datasetID}:
get:
tags:
- Datasets
summary: Retrieve a dataset by ID
description: Fetches details of a specific dataset within a repository by its unique identifier. This endpoint provides information about the dataset's metadata, labels, and other properties.
parameters:
- name: repositoryID
in: path
description: Unique identifier of the repository containing the dataset.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
- name: datasetID
in: path
description: Unique identifier of the dataset to be retrieved.
required: true
schema:
type: string
example: 3fa85f64-5717-4562-b3fc-2c963f66afa6
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/Dataset'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'404':
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'500':
description: INTERNAL SERVER ERROR
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
delete:
tags:
- Datasets
summary: Delete a dataset by ID
description: Permanently removes a dataset from the specified repository. This operation is irreversible and will delete all associated data points within the dataset.
parameters:
- name: repositoryID
in: path
description: Unique identifier of the repository containing the dataset to be deleted.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
- name: datasetID
in: path
description: Unique identifier of the dataset to delete.
required: true
schema:
type: string
example: 3fa85f64-5717-4562-b3fc-2c963f66afa6
responses:
'204':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'404':
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'500':
description: INTERNAL SERVER ERROR
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
patch:
tags:
- Datasets
summary: Update dataset metadata
description: Update the metadata fields of the latest version of a specified dataset within a repository. This operation allows partial updates to the dataset's metadata, including fields such as name, labels, totalDatapoints, version, and additional metadata.
parameters:
- name: repositoryID
in: path
description: Unique identifier of the repository containing the dataset to update.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
- name: datasetID
in: path
description: Unique identifier of the dataset to update.
required: true
schema:
type: string
example: 3fa85f64-5717-4562-b3fc-2c963f66afa6
requestBody:
description: JSON object containing the metadata fields to be updated for the dataset.
content:
application/json:
schema:
$ref: '#/components/schemas/DatasetMetadataUpdate'
required: true
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/Dataset'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDownload'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDownload'
'404':
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDownload'
'422':
description: UnprocessableEntity
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDownload'
'503':
description: Service Unavailable
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDownload'
/repositories/{repositoryID}/datasets/{datasetID}/datapoints:
get:
tags:
- Datasets
summary: Stream dataset data points
description: Streams data points of the specified dataset, providing real-time access to its content. This operation allows continuous retrieval of data points in a JSON format.
parameters:
- name: repositoryID
in: path
description: Unique identifier of the repository containing the dataset to stream.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
- name: datasetID
in: path
description: Unique identifier of the dataset from which data points will be streamed.
required: true
schema:
type: string
example: 3fa85f64-5717-4562-b3fc-2c963f66afa6
- name: version
in: query
description: Specific version of the dataset to stream. If omitted, the latest version will be streamed by default.
schema:
type: string
example: 3fa85f64-5717-40f3-8e13-77d862f722b5
- name: start
in: query
description: Starting index for streaming data points, used to control where the stream begins.
schema:
type: integer
example: 0
- name: size
in: query
description: Number of data points to retrieve in the stream, allowing control over the stream length.
schema:
type: integer
example: 100
responses:
'200':
description: Successful data point stream
headers:
Transfer-Encoding:
description: Indicates chunked transfer encoding for streaming response.
schema:
type: string
example: chunked
content:
application/json:
schema:
type: string
description: Streamed data points in JSON Lines format. Each line represents a JSON object containing individual data points.
example: '{"id": "0", "key": "value"}
{"id": "1", "key": "another_value"}
{"id": "2", "key": "more_data"}
'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'404':
description: Dataset or repository not found
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
put:
tags:
- Datasets
summary: Update dataset data points
description: Uploads new data points to update an existing dataset within the specified repository. This operation replaces the current data points with the new content provided in the request and generate a new version.
parameters:
- name: repositoryID
in: path
description: Unique identifier of the repository containing the dataset to be updated.
required: true
schema:
type: string
example: cb26237c-3beb-40f3-8e13-77d862f722b5
- name: datasetID
in: path
description: Unique identifier of the dataset from which data points will be streamed.
required: true
schema:
type: string
example: 3fa85f64-5717-4562-b3fc-2c963f66afa6
requestBody:
required: true
content:
multipart/form-data:
schema:
type: object
required:
- sourceData
properties:
sourceData:
type: string
description: A file containing the new dataset content, serialized in JSON Lines format. Each line represents a data point in JSON format.
format: binary
example: my_data.jsonl
responses:
'200':
description: OK
content:
application/json:
schema:
$ref: '#/components/schemas/Dataset'
'400':
description: Bad Request
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'401':
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'404':
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponseDataset'
'500':
description: INTERNAL SERVER ERROR
content:
application/json:
schema:
$ref: '#/components/schemas/HandleFailedResponse'
components:
schemas:
Dataset:
type: object
properties:
datasetId:
type: string
description: Unique identifier for the dataset.
example: 3fa85f64-5717-4562-b3fc-2c963f66afa6
repositoryId:
type: string
description: Unique identifier of the repository to which this dataset belongs.
example: cb26237c-3beb-40f3-8e13-77d862f722b5
name:
type: string
description: The name assigned to the dataset for easy identification.
example: dataset_name
labels:
type: array
description: List of labels associated with the dataset for categorization or tagging.
example:
- label
items:
type: string
metadata:
type: string
description: Additional metadata for the dataset in a key-value format.
example:
key: value
license:
type: string
description: The license of the uploaded dataset defines the permissions, restrictions, and obligations governing its use.
example:
id: OAL-1.0
url: https://github.com/Aleph-Alpha/.github/blob/main/OAL-1.0.pdf
version:
type: string
description: The current version of the dataset, allowing version tracking.
example: 55a62ef9-dd9c-4e5d-9b93-0361bae4db37
totalDatapoints:
type: integer
description: The total number of data points within the dataset.
example: 0
updatedAt:
type: string
description: The timestamp of the last update made to the dataset in ISO 8601 format.
example: '2024-07-04'
createdAt:
type: string
description: The timestamp when the dataset was initially created in ISO 8601 format.
example: '2024-07-04'
DatasetListResponse:
type: object
properties:
datasets:
type: array
description: A list of files retrieved for the current page.
items:
$ref: '#/components/schemas/Dataset'
page:
type: integer
description: The current page number in the paginated list.
example: 0
size:
type: integer
description: The number of datasets displayed per page.
example: 1
total:
type: integer
description: The total number of datasets available.
example: 1
HandleFailedResponseDataset:
type: object
properties:
message:
type: string
example: A descriptive error message based on the specific status code encountered, detailing the issue and possible causes.
DatasetMetadataUpdate:
type: object
properties:
labels:
type: array
description: List of labels associated with the dataset for categorization or tagging.
example:
- label
items:
type: string
metadata:
type: string
description: Additional metadata for the dataset in a key-value format.
example:
key: value
license:
type: string
description: The license of the uploaded dataset defines the permissions, restrictions, and obligations governing its use.
example:
id: OAL-1.0
url: https://github.com/Aleph-Alpha/.github/blob/main/OAL-1.0.pdf
name:
type: string
description: The name assigned to the dataset for easy identification.
example: dataset_name
totalDatapoints:
type: integer
description: The total number of data points within the dataset.
example: 0
HandleFailedResponseDownload:
type: object
properties:
message:
type: string
example: A descriptive error message based on the specific status code encountered, detailing the issue and possible causes.
HandleFailedResponse:
type: object
properties:
message:
type: string
example: A descriptive error message based on the specific status code encountered, detailing the issue and possible causes.
securitySchemes:
Bearer:
type: http
scheme: bearer
bearerFormat: JWT
x-tagGroups:
- name: PhariaData API
tags:
- Repositories
- Datasets
- Stages
- Downloads
- Transformations
- Connectors
- name: Pharia Search API
tags:
- Search Store
- Document