Amazon Data Pipeline website screenshot

Amazon Data Pipeline

AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals. With AWS Data Pipeline, you can regularly access your data where it is stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR. It supports data-driven workflows with retry, failure handling, and scheduling capabilities.

Amazon Data Pipeline publishes 4 APIs on the APIs.io network, including Pipeline Objects API, Pipeline Runs API, Pipelines API, and 1 more. Tagged areas include Data Processing, ETL, Workflows, Data Pipeline, and Automation.

The Amazon Data Pipeline catalog on APIs.io includes 1 JSON-LD context and 2 Spectral governance rulesets.

Amazon Data Pipeline’s developer surface includes authentication, developer portal, documentation, support, developer console, signup flow, and 23 more developer resources.

51.2/100 developing ▬ flat Agent 25/100 agent aware saas Full breakdown ↓
scored 2026-09-08 · rubric v0.20.0
AccessFreemiumSelf serve⚡ Free to try
1 APIs 7 Features 5 Use Cases
Data ProcessingETLWorkflowsData PipelineAutomation

Kin Score

Kin Score Kin Score How this is scored →
scored 2026-09-08 · rubric v0.20.0
Create-or-Update Ergonomics applies to this provider. This API accepts writes, so it carries 10 points of the composite. It is scored from the published contracts themselves: whether a caller can create-or-update in one call, whether the write accepts a key the caller already holds, and whether the response says which branch ran. Without that, every write needs a search-and-branch in front of it, and the first time that check is skipped a duplicate record is created. Scored against the observed mean rather than raw — a provider at the catalog average is unchanged by this facet, not penalised by it.
Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from github.com/api-evangelist/amazon-data-pipeline: open an issue to ask a question, or submit a pull request to add artifacts. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

APIs 4

Individual APIs this provider publishes, each with its own machine-readable definition.

Amazon Data Pipeline Pipeline Objects API

Operations for managing pipeline object definitions

Amazon Data Pipeline Pipeline Runs API

Operations for managing pipeline execution and task runs

Amazon Data Pipeline Pipelines API

Operations for managing data pipelines

Amazon Data Pipeline Tags API

Operations for managing pipeline tags

Postman Collections 1

Ready-to-run Postman collections for exercising this provider's APIs.

Open Collections 6

Open, tool-agnostic API collections (OpenAPI-derived and Bruno).

API Collection

OPEN COLLECTION

AWS Data Pipeline API

OPEN COLLECTION

Arazzo Workflows 9

Multi-step API workflows described with the Arazzo specification.

Amazon Data Pipeline Clone Pipeline

Copy an existing pipeline's definition into a brand-new pipeline and activate it.

ARAZZO

Amazon Data Pipeline Deactivate and Delete

Stop a running pipeline and then permanently remove it and its run history.

ARAZZO

Amazon Data Pipeline Export Definition

Confirm a pipeline exists and then export its active definition objects.

ARAZZO

Amazon Data Pipeline Inspect Running Tasks

Find running task instances in a pipeline and pull their full object definitions.

ARAZZO

Amazon Data Pipeline List and Describe

List all accessible pipelines and pull full metadata for the first page of them.

ARAZZO

Amazon Data Pipeline Provision and Activate

Create an empty pipeline, populate its definition, activate it, and confirm its state.

ARAZZO

Amazon Data Pipeline Redeploy Definition

Deactivate a pipeline, write a new definition, then reactivate it with the new objects.

ARAZZO

Amazon Data Pipeline Tag and Confirm

Add governance tags to a pipeline and confirm they are attached.

ARAZZO

Amazon Data Pipeline Validate Then Put Definition

Validate a candidate pipeline definition and only commit it when it is error free.

ARAZZO

Scroll for all 9

Pricing Plans 1

Published pricing tiers and plan structures.

Rate Limits 1

Documented rate limits and quota policies.

FinOps 1

Cost, billing, and metering signals for API financial operations.

Features 7

Notable capabilities this provider offers.

Data-Driven Workflows

Define complex data processing workflows with activities, data nodes, schedules, and preconditions using a declarative pipeline definition.

Multi-Service Integration

Move and transform data between Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift, and Amazon EMR in a single pipeline.

Flexible Scheduling

Schedule pipeline runs at fixed intervals (hourly, daily, weekly) or trigger them based on data availability preconditions.

Automated Retry and Failure Handling

Configure automatic retries for failed activities with configurable retry intervals, timeout settings, and failure notifications.

On-Premises Data Support

Process data from on-premises databases and file systems using the Data Pipeline Task Runner agent installed locally.

EMR Integration

Launch and manage Amazon EMR clusters as pipeline resources to run Hive, Pig, and MapReduce jobs as part of data workflows.

Pipeline Versioning

Manage active and latest pipeline definition versions, enabling updates to running pipelines without disrupting current execution.

Scroll for all 7

Semantic Vocabularies 1

JSON-LD contexts and semantic vocabularies used across these APIs.

Amazon Data Pipeline Context

0 classes · 30 properties

JSON-LD

Spectral Rules 2

Spectral governance rulesets for linting and validating these APIs.

Amazon Data Pipeline API Rules

5 rules · 3 warnings 2 info

SPECTRAL

Amazon Data Pipeline API Rules

26 rules · 13 errors 8 warnings 5 info

SPECTRAL

JSON Schema 16

Standalone JSON Schema definitions for this provider's data models.

Activate Pipeline Request

2 properties

JSON SCHEMA

Create Pipeline Output

1 properties

JSON SCHEMA

Create Pipeline Request

4 properties

JSON SCHEMA

Describe Pipelines Output

1 properties

JSON SCHEMA

Describe Pipelines Request

1 properties

JSON SCHEMA

Error

2 properties

JSON SCHEMA

Field

3 properties

JSON SCHEMA

Get Pipeline Definition Output

3 properties

JSON SCHEMA

List Pipelines Output

3 properties

JSON SCHEMA

Pipeline Description

4 properties

JSON SCHEMA

Pipeline ID Name

2 properties

JSON SCHEMA

Pipeline Object

3 properties

JSON SCHEMA

Put Pipeline Definition Output

3 properties

JSON SCHEMA

Query Objects Output

3 properties

JSON SCHEMA

Tag

2 properties

JSON SCHEMA

Validation Error

2 properties

JSON SCHEMA

Scroll for all 16

JSON Structure 16

JSON Structure definitions describing this provider's data shapes.

Activate Pipeline Request Structure

0 properties

JSON STRUCTURE

Create Pipeline Output Structure

0 properties

JSON STRUCTURE

Create Pipeline Request Structure

0 properties

JSON STRUCTURE

Describe Pipelines Output Structure

0 properties

JSON STRUCTURE

Describe Pipelines Request Structure

0 properties

JSON STRUCTURE

Error Structure

0 properties

JSON STRUCTURE

Field Structure

0 properties

JSON STRUCTURE

Get Pipeline Definition Output Structure

0 properties

JSON STRUCTURE

List Pipelines Output Structure

0 properties

JSON STRUCTURE

Pipeline Description Structure

0 properties

JSON STRUCTURE

Pipeline Id Name Structure

0 properties

JSON STRUCTURE

Pipeline Object Structure

0 properties

JSON STRUCTURE

Put Pipeline Definition Output Structure

0 properties

JSON STRUCTURE

Query Objects Output Structure

0 properties

JSON STRUCTURE

Tag Structure

0 properties

JSON STRUCTURE

Validation Error Structure

0 properties

JSON STRUCTURE

Scroll for all 16

Examples 16

Example request and response payloads for these APIs.

Error Example

2 fields

EXAMPLE

Field Example

2 fields

EXAMPLE

Pipeline Id Name Example

2 fields

EXAMPLE

Pipeline Object Example

3 fields

EXAMPLE

Tag Example

2 fields

EXAMPLE

Validation Error Example

2 fields

EXAMPLE

Scroll for all 16

Security Posture 4

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Amazon Data Pipeline Authentication

apiKey · 1 scheme

SECURITY

Amazon Data Pipeline Domain Security

TLSv1.3 · HSTS · DMARC

SECURITY

Amazon Data Pipeline Vulnerability Disclosure

security.txt · contact published

SECURITY

Amazon Data Pipeline Trust Center

PCI DSS, HIPAA, FedRAMP, GDPR, FIPS 140

SECURITY

Agentic Access 1

Recommended x-agentic-access execution contracts for AI agents.

Amazon Data Pipeline Agentic Access

13 operations · 13 acting

13 operations · 13 acting

AGENTIC

Use Cases 5

What developers build with this provider.

Daily ETL Workflows

Schedule daily extraction, transformation, and loading of data from relational databases into S3 or Redshift for analytics processing.

Log Processing Pipelines

Process application and server log files from S3 using EMR activities to generate aggregated reports and analytics datasets.

Database Migration

Migrate data between on-premises databases and AWS managed database services using scheduled pipeline activities.

Data Lake Ingestion

Automate the ingestion and transformation of raw data into structured formats in S3 data lakes for downstream analytics.

Cross-Region Data Replication

Replicate DynamoDB tables or S3 data across AWS regions using scheduled pipeline copy activities for disaster recovery.

Resources

Get Started 5

Portal, sign-up, and the first successful call

Documentation 1

Reference material describing how the API behaves

Agent Surfaces 1

MCP servers, agent skills, and machine-readable catalogs

Design & Contract 11

Pagination, idempotency, versioning, errors, and events

Scroll for all 11

Build 2

SDKs, sample code, and the tooling you integrate with

Access & Security 4

Authentication, authorization, and security posture

Operate 3

Status, limits, changes, and where to get help

Commercial 2

Pricing, plans, and the legal terms of use

Source (apis.yml)

apis.yml Raw ↑
aid: amazon-data-pipeline
name: Amazon Data Pipeline
description: AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute
  and storage services, as well as on-premises data sources, at specified intervals. With AWS Data Pipeline, you can regularly
  access your data where it is stored, transform and process it at scale, and efficiently transfer the results to AWS services
  such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR. It supports data-driven workflows with retry, failure handling,
  and scheduling capabilities.
type: Index
deliveryModel:
  model: saas
  open_source: false
  commercial: true
  callable_host: false
  label: Hosted service · you call their endpoint
  confidence: medium
  source:
  - openapi
  - pricing
  generated: '2026-08-28'
  method: derived
accessModel:
  pricing: freemium
  onboarding: self-serve
  trial: false
  try_now: true
  public: false
  label: Freemium · Self-serve signup
  confidence: medium
  source:
  - plans
  - authentication
  - security
  generated: '2026-09-03'
  method: derived
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/amazon-data-pipeline.png
tags:
- AWS
- Data Processing
- ETL
- Workflows
- Data Pipeline
- Automation
url: https://raw.githubusercontent.com/api-evangelist/amazon-data-pipeline/refs/heads/main/apis.yml
created: '2024-01-15'
modified: '2026-05-19'
specificationVersion: '0.23'
apis:
- aid: amazon-data-pipeline:amazon-data-pipeline-pipeline-objects-api
  name: Amazon Data Pipeline Pipeline Objects API
  description: Operations for managing pipeline object definitions
  humanURL: https://aws.amazon.com/datapipeline/
  baseURL: https://datapipeline.amazonaws.com
  tags:
  - Pipeline Objects
  properties:
  - type: OpenAPI
    url: openapi/amazon-data-pipeline-pipeline-objects-api-openapi.yml
  - type: Documentation
    url: https://docs.aws.amazon.com/datapipeline/
  - type: Pricing
    url: https://aws.amazon.com/datapipeline/pricing/
  - type: GettingStarted
    url: https://aws.amazon.com/datapipeline/getting-started/
  - type: FAQ
    url: https://aws.amazon.com/datapipeline/faqs/
  - type: APIReference
    url: https://docs.aws.amazon.com/datapipeline/latest/APIReference/
  - type: JSONSchema
    url: json-schema/pipeline-object-schema.json
  - type: JSONSchema
    url: json-schema/pipeline-description-schema.json
  - type: JSONLD
    url: json-ld/amazon-data-pipeline-context.jsonld
- aid: amazon-data-pipeline:amazon-data-pipeline-pipeline-runs-api
  name: Amazon Data Pipeline Pipeline Runs API
  description: Operations for managing pipeline execution and task runs
  humanURL: https://aws.amazon.com/datapipeline/
  baseURL: https://datapipeline.amazonaws.com
  tags:
  - Pipeline Runs
  properties:
  - type: OpenAPI
    url: openapi/amazon-data-pipeline-pipeline-runs-api-openapi.yml
  - type: Documentation
    url: https://docs.aws.amazon.com/datapipeline/
  - type: Pricing
    url: https://aws.amazon.com/datapipeline/pricing/
  - type: GettingStarted
    url: https://aws.amazon.com/datapipeline/getting-started/
  - type: FAQ
    url: https://aws.amazon.com/datapipeline/faqs/
  - type: APIReference
    url: https://docs.aws.amazon.com/datapipeline/latest/APIReference/
  - type: JSONSchema
    url: json-schema/pipeline-object-schema.json
  - type: JSONSchema
    url: json-schema/pipeline-description-schema.json
  - type: JSONLD
    url: json-ld/amazon-data-pipeline-context.jsonld
- aid: amazon-data-pipeline:amazon-data-pipeline-pipelines-api
  name: Amazon Data Pipeline Pipelines API
  description: Operations for managing data pipelines
  humanURL: https://aws.amazon.com/datapipeline/
  baseURL: https://datapipeline.amazonaws.com
  tags:
  - Pipelines
  properties:
  - type: OpenAPI
    url: openapi/amazon-data-pipeline-pipelines-api-openapi.yml
  - type: Documentation
    url: https://docs.aws.amazon.com/datapipeline/
  - type: Pricing
    url: https://aws.amazon.com/datapipeline/pricing/
  - type: GettingStarted
    url: https://aws.amazon.com/datapipeline/getting-started/
  - type: FAQ
    url: https://aws.amazon.com/datapipeline/faqs/
  - type: APIReference
    url: https://docs.aws.amazon.com/datapipeline/latest/APIReference/
  - type: JSONSchema
    url: json-schema/pipeline-object-schema.json
  - type: JSONSchema
    url: json-schema/pipeline-description-schema.json
  - type: JSONLD
    url: json-ld/amazon-data-pipeline-context.jsonld
- aid: amazon-data-pipeline:amazon-data-pipeline-tags-api
  name: Amazon Data Pipeline Tags API
  description: Operations for managing pipeline tags
  humanURL: https://aws.amazon.com/datapipeline/
  baseURL: https://datapipeline.amazonaws.com
  tags:
  - Tags
  properties:
  - type: OpenAPI
    url: openapi/amazon-data-pipeline-tags-api-openapi.yml
  - type: Documentation
    url: https://docs.aws.amazon.com/datapipeline/
  - type: Pricing
    url: https://aws.amazon.com/datapipeline/pricing/
  - type: GettingStarted
    url: https://aws.amazon.com/datapipeline/getting-started/
  - type: FAQ
    url: https://aws.amazon.com/datapipeline/faqs/
  - type: APIReference
    url: https://docs.aws.amazon.com/datapipeline/latest/APIReference/
  - type: JSONSchema
    url: json-schema/pipeline-object-schema.json
  - type: JSONSchema
    url: json-schema/pipeline-description-schema.json
  - type: JSONLD
    url: json-ld/amazon-data-pipeline-context.jsonld
common:
- type: AgenticAccess
  url: agentic-access/amazon-data-pipeline-agentic-access.yml
- type: TrustCenter
  url: security/amazon-data-pipeline-trust-center.yml
- type: VulnerabilityDisclosure
  url: security/amazon-data-pipeline-vulnerability-disclosure.yml
- type: DomainSecurity
  url: security/amazon-data-pipeline-domain-security.yml
- type: Authentication
  url: authentication/amazon-data-pipeline-authentication.yml
- type: PostmanWorkspace
  url: https://www.postman.com/kinlaneapi/amazon-data-pipeline/overview
- type: Arazzo
  url: arazzo/amazon-data-pipeline-clone-pipeline-workflow.yml
  name: Amazon Data Pipeline Clone Pipeline
- type: Arazzo
  url: arazzo/amazon-data-pipeline-deactivate-and-delete-workflow.yml
  name: Amazon Data Pipeline Deactivate and Delete
- type: Arazzo
  url: arazzo/amazon-data-pipeline-export-definition-workflow.yml
  name: Amazon Data Pipeline Export Definition
- type: Arazzo
  url: arazzo/amazon-data-pipeline-inspect-running-tasks-workflow.yml
  name: Amazon Data Pipeline Inspect Running Tasks
- type: Arazzo
  url: arazzo/amazon-data-pipeline-list-and-describe-workflow.yml
  name: Amazon Data Pipeline List and Describe
- type: Arazzo
  url: arazzo/amazon-data-pipeline-provision-and-activate-workflow.yml
  name: Amazon Data Pipeline Provision and Activate
- type: Arazzo
  url: arazzo/amazon-data-pipeline-redeploy-definition-workflow.yml
  name: Amazon Data Pipeline Redeploy Definition
- type: Arazzo
  url: arazzo/amazon-data-pipeline-tag-and-confirm-workflow.yml
  name: Amazon Data Pipeline Tag and Confirm
- type: Arazzo
  url: arazzo/amazon-data-pipeline-validate-then-put-definition-workflow.yml
  name: Amazon Data Pipeline Validate Then Put Definition
- type: Portal
  url: https://aws.amazon.com/datapipeline/
- type: DeveloperPortal
  url: https://aws.amazon.com/datapipeline/
- type: Documentation
  url: https://docs.aws.amazon.com/datapipeline/
- type: TermsOfService
  url: https://aws.amazon.com/service-terms/
- type: PrivacyPolicy
  url: https://aws.amazon.com/privacy/
- type: Support
  url: https://aws.amazon.com/premiumsupport/
- type: GitHubOrganization
  url: https://github.com/aws
- type: Console
  url: https://console.aws.amazon.com/datapipeline/
- type: Signup
  url: https://portal.aws.amazon.com/billing/signup
- type: Login
  url: https://signin.aws.amazon.com/
- type: StatusPage
  url: https://health.aws.amazon.com/health/status
- type: Contact
  url: https://aws.amazon.com/contact-us/
- type: SpectralRules
  url: rules/amazon-data-pipeline-spectral-rules.yml
- type: Vocabulary
  url: vocabulary/amazon-data-pipeline-vocabulary.yaml
- type: Features
  data:
  - name: Data-Driven Workflows
    description: Define complex data processing workflows with activities, data nodes, schedules, and preconditions using
      a declarative pipeline definition.
  - name: Multi-Service Integration
    description: Move and transform data between Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift, and Amazon EMR in
      a single pipeline.
  - name: Flexible Scheduling
    description: Schedule pipeline runs at fixed intervals (hourly, daily, weekly) or trigger them based on data availability
      preconditions.
  - name: Automated Retry and Failure Handling
    description: Configure automatic retries for failed activities with configurable retry intervals, timeout settings, and
      failure notifications.
  - name: On-Premises Data Support
    description: Process data from on-premises databases and file systems using the Data Pipeline Task Runner agent installed
      locally.
  - name: EMR Integration
    description: Launch and manage Amazon EMR clusters as pipeline resources to run Hive, Pig, and MapReduce jobs as part
      of data workflows.
  - name: Pipeline Versioning
    description: Manage active and latest pipeline definition versions, enabling updates to running pipelines without disrupting
      current execution.
- type: UseCases
  data:
  - name: Daily ETL Workflows
    description: Schedule daily extraction, transformation, and loading of data from relational databases into S3 or Redshift
      for analytics processing.
  - name: Log Processing Pipelines
    description: Process application and server log files from S3 using EMR activities to generate aggregated reports and
      analytics datasets.
  - name: Database Migration
    description: Migrate data between on-premises databases and AWS managed database services using scheduled pipeline activities.
  - name: Data Lake Ingestion
    description: Automate the ingestion and transformation of raw data into structured formats in S3 data lakes for downstream
      analytics.
  - name: Cross-Region Data Replication
    description: Replicate DynamoDB tables or S3 data across AWS regions using scheduled pipeline copy activities for disaster
      recovery.
- type: Integrations
  data:
  - name: Amazon S3
    description: Primary data node type for reading input data and writing output data in pipeline ETL activities using S3DataNode.
  - name: Amazon EMR
    description: Managed Hadoop/Spark cluster resource for running large-scale data processing activities including Hive,
      Pig, and MapReduce jobs.
  - name: Amazon RDS
    description: Relational database data node for SQL-based data extraction and loading between RDS instances and S3 or Redshift.
  - name: Amazon DynamoDB
    description: NoSQL data node for importing and exporting DynamoDB table data in pipeline activities for batch processing
      workflows.
  - name: Amazon Redshift
    description: Data warehouse target for loading processed pipeline output data for business intelligence and analytics
      queries.
  - name: AWS Glue
    description: Modern alternative managed ETL service that can complement or replace Data Pipeline for serverless data transformation
      workflows.
  - name: Amazon CloudWatch
    description: Monitor pipeline execution status, set up alarms for pipeline failures, and track activity completion metrics.
- type: Integrations
  url: https://aws.amazon.com/marketplace
integrations:
- name: Sign in
- name: Agent Mode
- name: Why AWS Marketplace?
- name: Get started in AWS Marketplace
- name: Industry
- name: Resources
- name: Become a Channel Partner
- name: Sell in AWS Marketplace
- name: Manage Your Account
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com
  url: https://apievangelist.com
x-parent: aws
x-relationship: product

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this
  • find_providersBrowse and filter every provider in the catalog.
  • get_provider_artifactsEvery artifact this provider publishes, grouped by type.
  • get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
  • get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
  • get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
  • get_provider_ratingPRO — composite, band, trend and facet scores.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This provider
curl "https://apis.io/api/v1/providers/amazon-data-pipeline"
All providers
curl "https://apis.io/api/v1/providers?limit=25"
Every operation they expose
curl "https://apis.io/api/v1/providers/amazon-data-pipeline/operations?limit=25"
How their score was established
curl "https://apis.io/api/v1/providers/amazon-data-pipeline/evidence"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

A second provider on the same verified email joins the account you already have.