A hosted extractor **job** represents the running extractor. Jobs produce logs and metrics that give the state of the job. For details on available states and metrics see documentation [here](https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors).
openapi: 3.1.0
info:
title: Cognite 3D Asset Mapping Jobs API
description: "# Introduction\nThis is the reference documentation for the Cognite API with\nan overview of all the available methods.\n\n# Postman\nSelect the **Download** button to download our OpenAPI specification to get started.\n\nTo import your data into Postman, select **Import**, and the Import modal opens.\nYou can import items by dragging or dropping files or folders. You can choose how to import your API and manage the import settings in **View Import Settings**.\n\nIn the Import Settings, set the **Folder organization** to **Tags**, select\n**Enable optional parameters** to turn off the settings, and select **Always inherit authentication** to turn on the settings. Select **Import**.\n\nSet the Authorization to **Oauth2.0**. By default, the settings are for Open Industrial Data. Navigate to [Cognite Hub](https://hub.cognite.com/open-industrial-data-211) to understand how to get the credentials for use in Postman.\n\nFor more information, see [Getting Started with Postman](https://developer.cognite.com/dev/guides/postman/).\n\n# Pagination\nMost resource types can be paginated, indicated by the field `nextCursor` in the response.\nBy passing the value of `nextCursor` as the cursor you will get the next page of `limit` results.\nNote that all parameters except `cursor` has to stay the same.\n\n# Parallel retrieval\nAs general guidance, Parallel Retrieval is a technique that should be used when due to query complexity, retrieval of data in a single request is significantly slower than it would otherwise be for a simple request. Parallel retrieval does not act as a speed multiplier on optimally running queries. By parallelizing such requests, data retrieval performance can be tuned to meet the client application needs. \n\nCDF supports parallel retrieval through the `partition` parameter, which has the format `m/n` where `n` is the amount of partitions you would like to split the entire data set into.\nIf you want to download the entire data set by splitting it into 10 partitions, do the following in parallel with `m` running from 1 to 10:\n - Make a request to `/events` with `partition=m/10`.\n - Paginate through the response by following the cursor as explained above. Note that the `partition` parameter needs to be passed to all subqueries.\n\nProcessing of parallel retrieval requests is subject to concurrency quota availability. The request returns the `429` response upon exceeding concurrency limits. See the Request throttling chapter below.\n\nTo prevent unexpected problems and to maximize read throughput, you should at most use 10 partitions. \nSome CDF resources will automatically enforce a maximum of 10 partitions.\nFor more specific and detailed information, please read the ```partition``` attribute documentation for the CDF resource you're using. \n\n# Requests throttling\nCognite Data Fusion (CDF) returns the HTTP `429` (too many requests) response status code when project capacity exceeds the limit.\n\nThe throttling can happen:\n - If a user or a project sends too many (more than allocated) concurrent requests.\n - If a user or a project sends a too high (more than allocated) rate of requests in a given amount of time.\n\nCognite recommends using a retry strategy based on truncated exponential backoff to handle sessions with HTTP response codes 429.\n\nCognite recommends using a reasonable number (up to 10) of `Parallel retrieval` partitions.\n\nFollowing these strategies lets you slow down the request frequency to maximize productivity without having to re-submit/retry failing requests.\n\nSee more [here](https://docs.cognite.com/dev/concepts/resource_throttling).\n\n# API versions\n## Version headers\nThis API uses calendar versioning, and version names follow the `YYYYMMDD` format.\nYou can find the versions currently available by using the version selector at the top of this page.\n\nTo use a specific API version, you can pass the `cdf-version: $version` header along with your requests to the API.\n\n## Beta versions\nThe beta versions provide a preview of what the stable version will look like in the future.\nBeta versions contain functionality that is reasonably mature, and highly likely to become a part of the stable API.\n\nBeta versions are indicated by a `-beta` suffix after the version name. For example, the beta version header for the\n2023-01-01 version is then `cdf-version: 20230101-beta`.\n\n## Alpha versions\nAlpha versions contain functionality that is new and experimental, and not guaranteed to ever become a part of the stable API.\nThis functionality presents no guarantee of service, so its use is subject to caution.\n\nAlpha versions are indicated by an `-alpha` suffix after the version name. For example, the alpha version header for\nthe 2023-01-01 version is then `cdf-version: 20230101-alpha`."
version: v1
contact:
name: Cognite Support
url: https://support.cognite.com
email: support@cognite.com
servers:
- url: https://{cluster}.cognitedata.com/api/v1/projects/{project}
description: The URL for the CDF cluster to connect to
variables:
cluster:
enum:
- api
- az-tyo-gp-001
- az-eastus-1
- az-power-no-northeurope
- westeurope-1
- asia-northeast1-1
- gc-dsm-gp-001
default: api
description: The CDF cluster to connect to
project:
default: publicdata
description: The CDF project name.
security:
- oidc-token:
- https://{cluster}.cognitedata.com/.default
- oauth2-client-credentials:
- https://{cluster}.cognitedata.com/.default
- oauth2-open-industrial-data:
- https://api.cognitedata.com/.default
- oauth2-auth-code:
- https://{cluster}.cognitedata.com/.default
tags:
- name: Jobs
description: A hosted extractor **job** represents the running extractor. Jobs produce logs and metrics that give the state of the job. For details on available states and metrics see documentation [here](https://docs.cognite.com/cdf/integration/guides/extraction/hosted_extractors).
paths:
/hostedextractors/jobs:
get:
tags:
- Jobs
operationId: list_jobs
x-capability:
- hostedExtractors:READ
description: '
> **Required capabilities:** `hostedExtractors:READ`
List all jobs in a given project. If more than `limit` jobs exist, a cursor for pagination will be returned with the response.'
parameters:
- $ref: '#/components/parameters/100Limit'
- $ref: '#/components/parameters/Cursor'
- in: query
name: source
description: External ID of source the returned jobs must be tied to.
required: false
schema:
type: string
title: Source
- in: query
name: destination
description: External ID of destination the returned jobs must be tied to.
required: false
schema:
type: string
title: Destination
- in: query
name: mapping
description: External ID of mapping the returned jobs must be tied to.
required: false
schema:
type: string
title: Mapping
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/ItemsWithCursor_MinimalJob_'
description: List of jobs and an optional cursor.
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: List Jobs
x-code-samples:
- lang: Python
label: Python SDK
source: "job_list = client.hosted_extractors.jobs.list(limit=5)\n\nfor job in client.hosted_extractors.jobs:\n job # do something with the job\n\nfor job_list in client.hosted_extractors.jobs(chunk_size=25):\n job_list # do something with the jobs\n"
post:
tags:
- Jobs
operationId: create_jobs
x-capability:
- hostedExtractors:READ
- hostedExtractors:WRITE
description: '
> **Required capabilities:** `hostedExtractors:READ` `hostedExtractors:WRITE`
Create up to 10 jobs.'
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/Items_CreateJob_'
required: true
description: Jobs to create.
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Items_MinimalJob_'
description: Successful Response
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: Create Jobs
x-code-samples:
- lang: Python
label: Python SDK
source: 'from cognite.client.data_classes.hosted_extractors import EventHubSourceWrite
job_write = EventHubSourceWrite(''my_event_hub'', ''http://myeventhub.com'', "My EventHub", ''my_key'', ''my_value'')
job = client.hosted_extractors.jobs.create(job_write)
'
/hostedextractors/jobs/byIds:
post:
tags:
- Jobs
operationId: retrieve_jobs
x-capability:
- hostedExtractors:READ
description: '
> **Required capabilities:** `hostedExtractors:READ`
Retrieve a list of up to 100 jobs by their external ID, optionally ignoring unknown IDs.'
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/ItemsWithIgnoreUnknownIds_ExternalId_'
required: true
description: List of external IDs of jobs to retrieve.
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Items_MinimalJob_'
description: List of retrieved jobs.
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: Retrieve Jobs
x-code-samples:
- lang: Python
label: Python SDK
source: 'res = client.hosted_extractors.jobs.retrieve(''myJob'')
res = client.hosted_extractors.jobs.retrieve(["myJob", "myOtherJob"], ignore_unknown_ids=True)
'
/hostedextractors/jobs/delete:
post:
tags:
- Jobs
operationId: delete_jobs
x-capability:
- hostedExtractors:WRITE
description: '
> **Required capabilities:** `hostedExtractors:WRITE`
Delete a list of jobs by their external ID.'
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/ItemsWithIgnoreUnknownIds_ExternalId_'
required: true
description: List of external IDs of jobs to delete.
responses:
'200':
$ref: '#/components/responses/EmptyResponse'
description: Empty response
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: Delete Jobs
x-code-samples:
- lang: Python
label: Python SDK
source: 'client.hosted_extractors.jobs.delete(["myMQTTJob", "MyEventHubJob"])
'
/hostedextractors/jobs/logs:
get:
tags:
- Jobs
operationId: get_job_logs
x-capability:
- hostedExtractors:READ
description: '
> **Required capabilities:** `hostedExtractors:READ`
List logs, optionally filtering on job, source, or destination. Logs are retrieved in reverse chronological order.'
parameters:
- in: query
name: job
required: false
schema:
type: string
title: Job
description: Require returned logs to belong to the job given by this external ID.
- $ref: '#/components/parameters/Limit'
- $ref: '#/components/parameters/Cursor'
- in: query
name: source
required: false
schema:
type: string
title: Source
description: Require returned logs to belong to the any job with source given by this external ID.
- in: query
name: destination
required: false
schema:
type: string
title: Destination
description: Require returned logs to belong to the any job with destination given by this external ID.
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Items_JobLogEntry_'
description: List of retrieved job logs
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: Get Job Logs
x-code-samples:
- lang: Python
label: Python SDK
source: 'res = client.hosted_extractors.jobs.list_logs(job="myJob")
'
/hostedextractors/jobs/metrics:
get:
tags:
- Jobs
operationId: get_job_metrics
x-capability:
- hostedExtractors:READ
description: '
> **Required capabilities:** `hostedExtractors:READ`
List metrics, optionally filtering on job, source, or destination. Logs are retrieved in reverse chronological order.'
parameters:
- in: query
name: job
required: false
schema:
type: string
title: Job
description: Require returned metrics to belong to the job given by this external ID.
- $ref: '#/components/parameters/Limit'
- $ref: '#/components/parameters/Cursor'
- in: query
name: source
required: false
schema:
type: string
title: Source
description: Require returned metrics to belong to the any job with source given by this external ID.
- in: query
name: destination
required: false
schema:
type: string
title: Destination
description: Require returned metrics to belong to the any job with destination given by this external ID.
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Items_JobMetrics_'
description: List of retrieved job metrics.
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: Get Job Metrics
x-code-samples:
- lang: Python
label: Python SDK
source: 'res = client.hosted_extractors.jobs.list_metrics(job="myJob")
'
/hostedextractors/jobs/update:
post:
tags:
- Jobs
operationId: update_jobs
x-capability:
- hostedExtractors:WRITE
description: '
> **Required capabilities:** `hostedExtractors:WRITE`
Update a list of up to 100 jobs.'
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/Items_JobUpdateItem_'
required: true
description: Jobs to update.
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/Items_MinimalJob_'
description: List of updated jobs.
'400':
$ref: '#/components/responses/400ErrorResponse'
'422':
$ref: '#/components/responses/ValidationError'
summary: Update Jobs
x-code-samples:
- lang: Python
label: Python SDK
source: 'from cognite.client.data_classes.hosted_extractors import EventHubSourceUpdate
job = EventHubSourceUpdate(''my_event_hub'').event_hub_name.set("My Updated EventHub")
updated_job = client.hosted_extractors.jobs.update(job)
'
components:
schemas:
CustomFormat:
additionalProperties: false
properties:
type:
enum:
- custom
title: Type
type: string
description: Format type.
mappingId:
type: string
title: MappingId
description: The ID of the mapping used for this format.
maxLength: 255
encoding:
$ref: '#/components/schemas/EncodingType'
compression:
$ref: '#/components/schemas/CompressionType'
expression:
type: string
title: Expression
description: The transformation expression of the mapping given by `mappingId`.
maxLength: 2000
required:
- type
- mappingId
- expression
title: CustomFormat
type: object
WriteCustomFormat:
additionalProperties: false
properties:
type:
enum:
- custom
title: Type
type: string
description: Format type.
encoding:
$ref: '#/components/schemas/EncodingType'
compression:
$ref: '#/components/schemas/CompressionType'
mappingId:
title: Mapping ID
type: string
description: ID of the mapping this format should be tied to.
maxLength: 255
required:
- type
- mappingId
title: WriteCustomFormat
type: object
description: Format the source data using a custom mapping.
CogniteExternalId:
description: The external ID provided by the client. Must be unique for the resource type.
type: string
maxLength: 255
example: my.known.id
DataModels:
type: object
additionalProperties: false
title: DataModels
description: Data models configuration to specify the space for all instances.
properties:
space:
type: string
title: Space
description: The data models space where time series will be created.
required:
- space
JobUpdateItem:
additionalProperties: false
properties:
externalId:
title: External ID
$ref: '#/components/schemas/CogniteExternalId'
update:
$ref: '#/components/schemas/JobUpdate'
required:
- externalId
- update
title: UpdateJob
type: object
ExternalIdWrapper:
additionalProperties: false
properties:
externalId:
$ref: '#/components/schemas/CogniteExternalId'
required:
- externalId
title: ExternalIdWrapper
type: object
KafkaJobConfig:
additionalProperties: false
type: object
title: Kafka
required:
- topic
properties:
topic:
title: Topic
type: string
description: Kafka topic to connect to
maxLength: 200
partitions:
title: Partitions
type: integer
minimum: 1
maximum: 10
default: 1
description: Number of partitions on the topic.
Error:
type: object
required:
- code
- message
description: Cognite API error.
properties:
code:
type: integer
description: HTTP status code.
format: int32
example: 401
message:
type: string
description: Error message.
example: Could not authenticate.
missing:
type: array
description: List of lookup objects that do not match any results.
items:
type: object
additionalProperties: true
duplicated:
type: array
description: List of objects that are not unique.
items:
type: object
additionalProperties: true
HeaderValueConfig:
additionalProperties: false
type: object
title: HeaderValueConfig
required:
- key
- value
- type
properties:
type:
title: Type
type: string
enum:
- headerValue
key:
title: Key
type: string
description: Key to insert the generated value into
value:
title: Value
type: string
description: Expression that will be evaluated, and its result used as a header value
JobLogEntry:
additionalProperties: false
properties:
jobExternalId:
type: string
description: External ID of the job this log entry belongs to.
maxLength: 255
type:
type: string
enum:
- paused
- startup_error
- connection_error
- connected
- transform_error
- cdf_write_error
- ok
description: 'Type of log entry.
`paused` indicates that the job has been stopped manually,
`startup_error` indicates that the job failed to start at all and requires changes to configuration,
`connected` indicates that the job has connected to the source but did not yet receive any data,
`transform_error` indicates that the job received data, but it failed to transform,
`cdf_write_error` indicates that ingesting the data to CDF failed,
`ok` means that data was successfully ingested to CDF.'
message:
type: string
description: Log message. Not all log entries have messages.
createdTime:
$ref: '#/components/schemas/EpochTimestamp'
title: JobLogEntry
type: object
required:
- jobExternalId
- type
- createdTime
RestJobConfig:
additionalProperties: false
type: object
title: Rest
properties:
interval:
$ref: '#/components/schemas/RestInterval'
path:
type: string
title: Path
description: Path of resource to access on the server, without query.
minLength: 1
maxLength: 2048
method:
type: string
title: Method
description: HTTP method to use for each request.
enum:
- get
- post
default: get
body:
title: Body
description: 'Initial JSON body to send with request. Only applicable if method is `post`. Maximum of 10000 bytes total.
'
query:
type: object
title: Query
description: 'Query parameters to include in request. String key -> String value. Limits: Maximum 255 characters per key, 2048 per value, and at most 32 pairs.
'
x-maxKeyLength: 255
maxProperties: 32
additionalProperties:
type: string
maxLength: 2048
headers:
type: object
title: Query
description: 'Headers to include in request. String key -> String value. Limits: Maximum 255 characters per key, 2048 per value, and at most 32 pairs.
'
x-maxKeyLength: 255
maxProperties: 32
additionalProperties:
type: string
maxLength: 2048
incrementalLoad:
type: object
discriminator:
mapping:
queryParam: '#/components/schemas/QueryParameterConfig'
headerValue: '#/components/schemas/HeaderValueConfig'
body: '#/components/schemas/BodyPaginationConfig'
propertyName: type
oneOf:
- $ref: '#/components/schemas/QueryParameterConfig'
- $ref: '#/components/schemas/HeaderValueConfig'
- $ref: '#/components/schemas/BodyPaginationConfig'
title: Incremental load
description: The format of the messages from the source. This is used to convert messages coming from the source system to a format that can be inserted into CDF.
pagination:
type: object
discriminator:
mapping:
queryParam: '#/components/schemas/QueryParameterConfig'
headerValue: '#/components/schemas/HeaderValueConfig'
nextUrl: '#/components/schemas/NextUrlConfig'
body: '#/components/schemas/BodyPaginationConfig'
propertyName: type
oneOf:
- $ref: '#/components/schemas/QueryParameterConfig'
- $ref: '#/components/schemas/HeaderValueConfig'
- $ref: '#/components/schemas/NextUrlConfig'
- $ref: '#/components/schemas/BodyPaginationConfig'
title: Incremental load
description: The format of the messages from the source. This is used to convert messages coming from the source system to a format that can be inserted into CDF.
required:
- interval
- path
UpdateItem_DestinationId_:
additionalProperties: false
properties:
set:
title: Set
type: string
maxLength: 255
required:
- set
title: UpdateItem_DestinationId
type: object
description: Set a new destination this job should write to.
Items_JobLogEntry_:
additionalProperties: false
properties:
items:
items:
$ref: '#/components/schemas/JobLogEntry'
title: Items
type: array
maxLength: 1000
minLength: 0
required:
- items
title: JobLogEntryList
type: object
ItemsWithCursor_MinimalJob_:
additionalProperties: false
properties:
items:
items:
$ref: '#/components/schemas/MinimalJob'
title: Items
type: array
maxLength: 100
minLength: 0
nextCursor:
type: string
title: Next cursor
description: Cursor for pagination
required:
- items
title: JobList
type: object
NextUrlConfig:
additionalProperties: false
type: object
title: HeaderValueConfig
required:
- type
- value
properties:
type:
title: Type
type: string
enum:
- nextUrl
value:
title: Value
type: string
description: Expression yielding the next URL to call
MQTTJobConfig:
additionalProperties: false
type: object
title: MQTT
required:
- topicFilter
properties:
topicFilter:
title: TopicFilter
type: string
description: Topic filter.
maxLength: 200
PrefixConfig:
additionalProperties: false
properties:
fromTopic:
type: boolean
title: FromTopic
description: Generate the prefix based on the topic of the received message.
prefix:
type: string
title: Prefix
description: A fixed prefix to the generated IDs.
maxLength: 255
title: PrefixConfig
description: Generate a prefix for resources created using this format. If both `prefix` and `fromTopic` are set, the generated ID will be on the form `[prefix][topic][id]`
type: object
EpochTimestamp:
description: The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
type: integer
minimum: 0
format: int64
example: 1730204346000
UpdateFormat:
additionalProperties: false
properties:
set:
discriminator:
mapping:
cognite: '#/components/schemas/CogniteFormat'
custom: '#/components/schemas/WriteCustomFormat'
rockwell: '#/components/schemas/RockwellFormat'
value: '#/components/schemas/ValueFormat'
propertyName: type
oneOf:
- $ref: '#/components/schemas/RockwellFormat'
- $ref: '#/components/schemas/CogniteFormat'
- $ref: '#/components/schemas/WriteCustomFormat'
- $ref: '#/components/schemas/ValueFormat'
title: Set
description: The format of the messages from the source. This is used to convert messages coming from the source system to a format that can be inserted into CDF.
type: object
required:
- set
title: UpdateFormat
type: object
description: Set a new format.
ValueFormat:
additionalProperties: false
properties:
type:
enum:
- value
title: Type
type: string
description: Format type.
encoding:
$ref: '#/components/schemas/EncodingType'
compression:
$ref: '#/components/schemas/CompressionType'
prefix:
$ref: '#/components/schemas/PrefixConfig'
dataModels:
$ref: '#/components/schemas/DataModels'
required:
- type
title: ValueFormat
type: object
description: 'Assume the source data is on the form `some string` or `123.456`, and convert these into CDF datapoints, using prefix config to determine the timeseries external ID, and current time as timestamp.
If no prefix config is set, use the topic name as timeseries ID.'
UpdateItem_JobTargetStatus_:
additionalProperties: false
properties:
set:
$ref: '#/components/schemas/JobTargetStatus'
required:
- set
title: UpdateItem_JobTargetStatus
type: object
description: Set a new job target status.
JobMetrics:
additionalProperties: false
properties:
jobExternalId:
type: string
description: External ID of the job this metrics batch belongs to.
maxLength: 255
timestamp:
title: Timestamp
description: 'The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
Metrics are from the UTC hour this timestamp is ingest. For example, if this timestamp is at 01:43:15, the metrics batch contains metrics from 01:00:00 to 01:43:15.'
type: integer
minimum: 0
format: int64
sourceMessages:
title: SourceMessages
minimum: 0
format: int64
type: integer
description: Messages received from the source.
cdfInputValues:
title: CDFInputValues
minimum: 0
format: int64
type: integer
description: Destination resources successfully transformed and passed to CDF.
cdfRequests:
title: CDFRequests
minimum: 0
format: int64
type: integer
description: Requests made to CDF containing data produced by this job.
transformFailures:
title: TransformFailures
minimum: 0
format: int64
type: integer
description: Source messages that failed to transform.
cdfWriteFailures:
title: CDFWriteFailures
minimum: 0
format: int64
type: integer
description: Times the destination received data from transformations, but failed to produce a valid request to CDF.
cdfSkippedValues:
title: CDFSkippedValues
minimum: 0
format: int64
type: integer
description: Values the destination received from the source, then decided to skip due to data type mismatch, invalid content, or other.
cdfFailedValues:
title: CDFFailedValues
minimum: 0
format: int64
type: integer
description: Values the destination was unable to upload to CDF.
cdfUploadedValues:
title: CDFUploadedValues
minimum: 0
format: int64
type: integer
description: Values the destination successfully uploaded to CDF.
required:
- jobExternalId
- timestamp
- sourceMessages
- cdfInputValues
- cdfRequests
- transformFailures
- cdfWriteFailures
- cdfSkippedValues
- cdfFailedValues
- cdfUploadedValues
CreateJob:
additionalProperties: false
properties:
externalId:
$ref: '#/components/schemas/CogniteExternalId'
destinationId:
title: Destination ID
description: ID of the destination this job should write to.
type: string
maxLength: 255
sourceId:
title: Source ID
description: ID of the source this job should read from.
type: string
maxLen
# --- truncated at 32 KB (49 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/cognite/refs/heads/main/openapi/cognite-jobs-api-openapi.yml