Amazon Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals. With AWS Data Pipeline, you can regularly access your data where it is stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR. It supports data-driven workflows with retry, failure handling, and scheduling capabilities.
Amazon Data Pipeline publishes 4 APIs on the APIs.io network, including Pipeline Objects API, Pipeline Runs API, Pipelines API, and 1 more. Tagged areas include Data Processing, ETL, Workflows, Data Pipeline, and Automation.
The Amazon Data Pipeline catalog on APIs.io includes 1 JSON-LD context and 2 Spectral governance rulesets.
Amazon Data Pipeline’s developer surface includes authentication, developer portal, documentation, support, developer console, signup flow, and 23 more developer resources.
Kin Score
Kin Score
How this is scored →scored 2026-07-28 · rubric v0.6
APIs 4
Individual APIs this provider publishes, each with its own machine-readable definition.
Amazon Data Pipeline Pipeline Objects API
Operations for managing pipeline object definitions
Amazon Data Pipeline Pipeline Runs API
Operations for managing pipeline execution and task runs
Amazon Data Pipeline Pipelines API
Operations for managing data pipelines
Amazon Data Pipeline Tags API
Operations for managing pipeline tags
Postman Collections 1
Ready-to-run Postman collections for exercising this provider's APIs.
AWS Data Pipeline API
POSTMANOpen Collections 1
Open, tool-agnostic API collections (OpenAPI-derived and Bruno).
AWS Data Pipeline API
OPEN COLLECTIONArazzo Workflows 9
Multi-step API workflows described with the Arazzo specification.
Amazon Data Pipeline Clone Pipeline
Copy an existing pipeline's definition into a brand-new pipeline and activate it.
ARAZZOAmazon Data Pipeline Deactivate and Delete
Stop a running pipeline and then permanently remove it and its run history.
ARAZZOAmazon Data Pipeline Export Definition
Confirm a pipeline exists and then export its active definition objects.
ARAZZOAmazon Data Pipeline Inspect Running Tasks
Find running task instances in a pipeline and pull their full object definitions.
ARAZZOAmazon Data Pipeline List and Describe
List all accessible pipelines and pull full metadata for the first page of them.
ARAZZOAmazon Data Pipeline Provision and Activate
Create an empty pipeline, populate its definition, activate it, and confirm its state.
ARAZZOAmazon Data Pipeline Redeploy Definition
Deactivate a pipeline, write a new definition, then reactivate it with the new objects.
ARAZZOAmazon Data Pipeline Tag and Confirm
Add governance tags to a pipeline and confirm they are attached.
ARAZZOAmazon Data Pipeline Validate Then Put Definition
Validate a candidate pipeline definition and only commit it when it is error free.
ARAZZOScroll for all 9
Pricing Plans 1
Published pricing tiers and plan structures.
Rate Limits 1
Documented rate limits and quota policies.
Amazon Data Pipeline Rate Limits
RATE LIMITSFinOps 1
Cost, billing, and metering signals for API financial operations.
Features 7
Notable capabilities this provider offers.
Data-Driven Workflows
Define complex data processing workflows with activities, data nodes, schedules, and preconditions using a declarative pipeline definition.
Multi-Service Integration
Move and transform data between Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift, and Amazon EMR in a single pipeline.
Flexible Scheduling
Schedule pipeline runs at fixed intervals (hourly, daily, weekly) or trigger them based on data availability preconditions.
Automated Retry and Failure Handling
Configure automatic retries for failed activities with configurable retry intervals, timeout settings, and failure notifications.
On-Premises Data Support
Process data from on-premises databases and file systems using the Data Pipeline Task Runner agent installed locally.
EMR Integration
Launch and manage Amazon EMR clusters as pipeline resources to run Hive, Pig, and MapReduce jobs as part of data workflows.
Pipeline Versioning
Manage active and latest pipeline definition versions, enabling updates to running pipelines without disrupting current execution.
Scroll for all 7
Semantic Vocabularies 1
JSON-LD contexts and semantic vocabularies used across these APIs.
Amazon Data Pipeline Context
JSON-LDSpectral Rules 2
Spectral governance rulesets for linting and validating these APIs.
Amazon Data Pipeline API Rules
SPECTRALAmazon Data Pipeline API Rules
SPECTRALJSON Schema 16
Standalone JSON Schema definitions for this provider's data models.
Activate Pipeline Request
JSON SCHEMACreate Pipeline Output
JSON SCHEMACreate Pipeline Request
JSON SCHEMADescribe Pipelines Output
JSON SCHEMADescribe Pipelines Request
JSON SCHEMAError
JSON SCHEMAField
JSON SCHEMAGet Pipeline Definition Output
JSON SCHEMAList Pipelines Output
JSON SCHEMAPipeline Description
JSON SCHEMAPipeline ID Name
JSON SCHEMAPipeline Object
JSON SCHEMAPut Pipeline Definition Output
JSON SCHEMAQuery Objects Output
JSON SCHEMATag
JSON SCHEMAValidation Error
JSON SCHEMAScroll for all 16
JSON Structure 16
JSON Structure definitions describing this provider's data shapes.
Activate Pipeline Request Structure
JSON STRUCTURECreate Pipeline Output Structure
JSON STRUCTURECreate Pipeline Request Structure
JSON STRUCTUREDescribe Pipelines Output Structure
JSON STRUCTUREDescribe Pipelines Request Structure
JSON STRUCTUREError Structure
JSON STRUCTUREField Structure
JSON STRUCTUREGet Pipeline Definition Output Structure
JSON STRUCTUREList Pipelines Output Structure
JSON STRUCTUREPipeline Description Structure
JSON STRUCTUREPipeline Id Name Structure
JSON STRUCTUREPipeline Object Structure
JSON STRUCTUREPut Pipeline Definition Output Structure
JSON STRUCTUREQuery Objects Output Structure
JSON STRUCTURETag Structure
JSON STRUCTUREValidation Error Structure
JSON STRUCTUREScroll for all 16
Examples 16
Example request and response payloads for these APIs.
Error Example
EXAMPLEField Example
EXAMPLEPipeline Description Example
EXAMPLEPipeline Id Name Example
EXAMPLEPipeline Object Example
EXAMPLEQuery Objects Output Example
EXAMPLETag Example
EXAMPLEValidation Error Example
EXAMPLEScroll for all 16
Security Posture 4
Authentication, domain security, vulnerability disclosure, and trust-center signals.
Agentic Access 1
Recommended x-agentic-access execution contracts for AI agents.
Use Cases 5
What developers build with this provider.
Daily ETL Workflows
Schedule daily extraction, transformation, and loading of data from relational databases into S3 or Redshift for analytics processing.
Log Processing Pipelines
Process application and server log files from S3 using EMR activities to generate aggregated reports and analytics datasets.
Database Migration
Migrate data between on-premises databases and AWS managed database services using scheduled pipeline activities.
Data Lake Ingestion
Automate the ingestion and transformation of raw data into structured formats in S3 data lakes for downstream analytics.
Cross-Region Data Replication
Replicate DynamoDB tables or S3 data across AWS regions using scheduled pipeline copy activities for disaster recovery.
Resources
Get Started 5
Portal, sign-up, and the first successful call
Documentation 1
Reference material describing how the API behaves
Agent Surfaces 1
MCP servers, agent skills, and machine-readable catalogs
Design & Contract 11
Pagination, idempotency, versioning, errors, and events
Scroll for all 11
Build 2
SDKs, sample code, and the tooling you integrate with
Access & Security 4
Authentication, authorization, and security posture
Operate 3
Status, limits, changes, and where to get help
Commercial 2
Pricing, plans, and the legal terms of use