Cloud Storage and Data Acquisition

aid: cloud-storage-and-data-acquisition
name: Cloud Storage and Data Acquisition
description: Cloud Storage and Data Acquisition is a topic profile in the API Evangelist Network covering APIs and tooling
  for ingesting, moving, and persisting bulk and streaming data into cloud-resident storage. It groups object storage services,
  data-lake foundations, managed ingestion pipelines, change-data-capture connectors, transfer appliances, and data-broker
  APIs that provide source material for cloud-storage workloads. The topic is intended as an entry point for developers and
  architects evaluating how data lands in cloud storage from on-premises systems, SaaS APIs, IoT devices, public data sources,
  and partner exchanges.
type: Index
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
url: https://raw.githubusercontent.com/api-evangelist/cloud-storage-and-data-acquisition/refs/heads/main/apis.yml
created: '2024-01-01'
modified: '2026-04-23'
specificationVersion: '0.19'
kind: topic
tags:
- Bulk Transfer
- Change Data Capture
- Cloud Storage
- Data Acquisition
- Data Ingestion
- Data Lake
- ETL
- Object Storage
- Pipelines
- Streaming
apis:
- aid: cloud-storage-and-data-acquisition:object-storage
  name: Object Storage Surface
  tags:
  - Object Storage
  - REST
  image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
  humanURL: https://aws.amazon.com/s3/
  properties:
  - url: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html
    type: Documentation
  - url: https://cloud.google.com/storage/docs/json_api/v1
    type: Documentation
  - url: https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-rest-api
    type: Documentation
  description: The object-storage surface includes Amazon S3, Google Cloud Storage, and Azure Blob Storage REST APIs. These
    APIs provide the canonical landing zone for cloud data acquisition and are the most common targets for ingestion pipelines.
- aid: cloud-storage-and-data-acquisition:streaming-ingest
  name: Streaming Ingest Surface
  tags:
  - Pub/Sub
  - Real-time
  - Streaming
  image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
  humanURL: https://aws.amazon.com/kinesis/data-streams/
  properties:
  - url: https://docs.aws.amazon.com/kinesis/latest/APIReference/
    type: Documentation
  - url: https://cloud.google.com/pubsub/docs/reference/rest
    type: Documentation
  - url: https://learn.microsoft.com/en-us/rest/api/eventhub/
    type: Documentation
  description: The streaming-ingest surface covers Amazon Kinesis Data Streams, Google Cloud Pub/Sub, and Azure Event Hubs.
    These services ingest high-volume event streams and make them durable for downstream landing into object storage and analytics
    systems.
- aid: cloud-storage-and-data-acquisition:managed-ingestion
  name: Managed Ingestion Pipelines
  tags:
  - Data Pipelines
  - ETL
  - Managed
  image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
  humanURL: https://aws.amazon.com/glue/
  properties:
  - url: https://docs.aws.amazon.com/glue/latest/webapi/Welcome.html
    type: Documentation
  - url: https://cloud.google.com/dataflow/docs/reference/rest
    type: Documentation
  - url: https://learn.microsoft.com/en-us/rest/api/datafactory/
    type: Documentation
  description: Managed ingestion pipelines such as AWS Glue, Google Cloud Dataflow, and Azure Data Factory expose REST APIs
    for orchestrating extract-transform-load and extract-load-transform jobs that land source data in cloud storage.
- aid: cloud-storage-and-data-acquisition:change-data-capture
  name: Change Data Capture Connectors
  tags:
  - CDC
  - Connectors
  - Replication
  image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
  humanURL: https://debezium.io/
  properties:
  - url: https://debezium.io/documentation/
    type: Documentation
  - url: https://docs.aws.amazon.com/dms/
    type: Documentation
  description: Change Data Capture connectors (Debezium, AWS DMS, Fivetran, Striim) replicate row-level changes from operational
    databases to cloud storage and warehouses, providing a low-latency feed for analytics and data-lake hydration.
- aid: cloud-storage-and-data-acquisition:transfer-services
  name: Bulk Transfer Services
  tags:
  - Bulk Transfer
  - Migration
  - Offline
  image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
  humanURL: https://aws.amazon.com/datasync/
  properties:
  - url: https://docs.aws.amazon.com/datasync/latest/userguide/API_Reference.html
    type: Documentation
  - url: https://cloud.google.com/storage-transfer/docs/reference/rest
    type: Documentation
  - url: https://learn.microsoft.com/en-us/azure/databox/
    type: Documentation
  description: Bulk transfer services (AWS DataSync and Snow family, Google Storage Transfer Service, Azure Data Box) move
    large datasets from on-premises and edge locations into cloud storage over network or via offline appliances.
- aid: cloud-storage-and-data-acquisition:data-marketplaces
  name: Data Marketplaces
  tags:
  - Datasets
  - Marketplace
  - Open Data
  image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
  humanURL: https://aws.amazon.com/data-exchange/
  properties:
  - url: https://aws.amazon.com/data-exchange/
    type: Documentation
  - url: https://cloud.google.com/marketplace
    type: Documentation
  - url: https://registry.opendata.aws/
    type: Open Data Registry
  description: Data marketplaces and open-data registries expose third-party datasets through REST APIs, subscription delivery,
    and shared buckets. They are an increasingly common acquisition channel for cloud-storage data lakes.
common:
- type: Topic
  url: https://apievangelist.com/topics/cloud-storage-and-data-acquisition/
- type: API Evangelist
  url: https://apievangelist.com/
- type: Network
  url: https://network.apievangelist.com/
- type: GitHub
  url: https://github.com/api-evangelist
- type: JSONLD
  url: json-ld/cloud-storage-and-data-acquisition-context.jsonld
- type: Spectral
  url: rules/cloud-storage-and-data-acquisition-rules.yml
maintainers:
- FN: Kin Lane
  email: kinlane@gmail.com