Amazon Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals. With AWS Data Pipeline, you can regularly access your data where it is stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR. It supports data-driven workflows with retry, failure handling, and scheduling capabilities.
APIs
AWS Data Pipeline API
The AWS Data Pipeline API provides a web service for processing and moving data between different AWS compute and storage services as well as on-premises data sources at specifi...
Capabilities
Features
Define complex data processing workflows with activities, data nodes, schedules, and preconditions using a declarative pipeline definition.
Move and transform data between Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift, and Amazon EMR in a single pipeline.
Schedule pipeline runs at fixed intervals (hourly, daily, weekly) or trigger them based on data availability preconditions.
Configure automatic retries for failed activities with configurable retry intervals, timeout settings, and failure notifications.
Process data from on-premises databases and file systems using the Data Pipeline Task Runner agent installed locally.
Launch and manage Amazon EMR clusters as pipeline resources to run Hive, Pig, and MapReduce jobs as part of data workflows.
Manage active and latest pipeline definition versions, enabling updates to running pipelines without disrupting current execution.
Use Cases
Schedule daily extraction, transformation, and loading of data from relational databases into S3 or Redshift for analytics processing.
Process application and server log files from S3 using EMR activities to generate aggregated reports and analytics datasets.
Migrate data between on-premises databases and AWS managed database services using scheduled pipeline activities.
Automate the ingestion and transformation of raw data into structured formats in S3 data lakes for downstream analytics.
Replicate DynamoDB tables or S3 data across AWS regions using scheduled pipeline copy activities for disaster recovery.
Integrations
Primary data node type for reading input data and writing output data in pipeline ETL activities using S3DataNode.
Managed Hadoop/Spark cluster resource for running large-scale data processing activities including Hive, Pig, and MapReduce jobs.
Relational database data node for SQL-based data extraction and loading between RDS instances and S3 or Redshift.
NoSQL data node for importing and exporting DynamoDB table data in pipeline activities for batch processing workflows.
Data warehouse target for loading processed pipeline output data for business intelligence and analytics queries.
Modern alternative managed ETL service that can complement or replace Data Pipeline for serverless data transformation workflows.
Monitor pipeline execution status, set up alarms for pipeline failures, and track activity completion metrics.