Streaming

aid: streaming
name: Streaming
description: 'Streaming is a topic catalog of the protocols, platforms, and processing engines used to move and transform
  real-time, high-volume, often bidirectional data over the network. It indexes the canonical log-structured and broker systems
  (Apache Kafka, Apache Pulsar, Redpanda, NATS JetStream, AWS Kinesis, GCP Pub/Sub + Dataflow, Azure Event Hubs, Confluent
  Cloud, StreamNative), the over-the-wire streaming protocols exposed to API consumers (Server-Sent Events, WebSocket, gRPC
  streaming, GraphQL subscriptions), the change-data capture and connector frameworks that feed them (Kafka Connect, Debezium),
  and the stream-processing engines that consume them (Apache Flink, Spark Structured Streaming, Materialize, Tinybird, Bytewax,
  Apache Beam). This topic is distinguished from `events` and `async-apis`: streaming emphasizes real-time, high-throughput,
  partitioned, and often bidirectional pipes, rather than discrete event envelopes or static contract documents.'
type: Index
position: Consumer
access: 3rd-Party
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg
tags:
- Streaming
- Real Time
- Event Streaming
- Change Data Capture
- Stream Processing
- Server Sent Events
- WebSocket
- gRPC
- GraphQL Subscriptions
- Kafka
- Pulsar
- Kinesis
- Flink
created: '2026-05-22'
modified: '2026-05-22'
url: https://raw.githubusercontent.com/api-evangelist/streaming/refs/heads/main/apis.yml
specificationVersion: '0.19'
apis:
- aid: streaming:apache-kafka
  name: Apache Kafka
  description: Distributed, partitioned, replicated log. The reference open-source streaming platform; durable, ordered topics
    with consumer groups, exactly -once semantics, and the de facto wire protocol for the streaming ecosystem. Native Kafka
    clients and a REST proxy expose the broker.
  humanURL: https://kafka.apache.org
  tags:
  - Streaming
  - Log
  - Open Source
  - Apache Software Foundation
  properties:
  - type: Documentation
    url: https://kafka.apache.org/documentation/
  - type: GitHubRepository
    url: https://github.com/apache/kafka
  - type: Topic
    url: https://github.com/api-evangelist/apache-kafka
- aid: streaming:apache-pulsar
  name: Apache Pulsar
  description: Cloud-native, multi-tenant pub/sub and streaming platform with a tiered storage architecture (BookKeeper) that
    separates compute from storage, native geo-replication, and built-in Functions for lightweight stream processing.
  humanURL: https://pulsar.apache.org
  tags:
  - Streaming
  - Pub Sub
  - Open Source
  - Apache Software Foundation
  properties:
  - type: Documentation
    url: https://pulsar.apache.org/docs/
  - type: GitHubRepository
    url: https://github.com/apache/pulsar
  - type: Topic
    url: https://github.com/api-evangelist/apache-pulsar
- aid: streaming:redpanda
  name: Redpanda
  description: Kafka-API-compatible streaming platform implemented in C++ with no ZooKeeper/JVM dependency. Single binary,
    thread-per-core architecture, Raft consensus; positioned as a drop-in for Kafka workloads.
  humanURL: https://redpanda.com
  tags:
  - Streaming
  - Kafka Compatible
  - Open Source
  properties:
  - type: Documentation
    url: https://docs.redpanda.com
  - type: GitHubRepository
    url: https://github.com/redpanda-data/redpanda
- aid: streaming:nats-jetstream
  name: NATS JetStream
  description: Persistence layer for the NATS messaging system providing at-least-once and exactly-once streaming, key/value
    and object stores, and durable consumers — designed for edge, IoT, and microservice topologies.
  humanURL: https://nats.io/
  tags:
  - Streaming
  - Messaging
  - Open Source
  - CNCF
  properties:
  - type: Documentation
    url: https://docs.nats.io/nats-concepts/jetstream
  - type: GitHubRepository
    url: https://github.com/nats-io/nats-server
- aid: streaming:aws-kinesis
  name: Amazon Kinesis
  description: 'AWS managed family for real-time streaming: Kinesis Data Streams (shards, partition keys, 24h–365d retention),
    Kinesis Data Firehose (delivery to S3/Redshift/OpenSearch), and Kinesis Video Streams for media. HTTP/2 SubscribeToShard
    for low-latency consumers.'
  humanURL: https://aws.amazon.com/kinesis/
  tags:
  - Streaming
  - AWS
  - Managed
  properties:
  - type: Documentation
    url: https://docs.aws.amazon.com/kinesis/
  - type: Topic
    url: https://github.com/api-evangelist/amazon-kinesis
- aid: streaming:gcp-pubsub
  name: Google Cloud Pub/Sub and Dataflow
  description: GCP's managed messaging (Pub/Sub) and stream-processing (Dataflow, built on Apache Beam) stack. Pub/Sub provides
    at-least-once and exactly-once delivery with push/pull subscribers; Dataflow runs windowed, watermark-aware Beam pipelines.
  humanURL: https://cloud.google.com/pubsub
  tags:
  - Streaming
  - GCP
  - Managed
  properties:
  - type: Documentation
    url: https://cloud.google.com/pubsub/docs
  - type: Documentation
    url: https://cloud.google.com/dataflow/docs
- aid: streaming:azure-event-hubs
  name: Azure Event Hubs
  description: Microsoft's managed big-data streaming platform; Kafka-protocol compatible, partitioned, with Capture (delivery
    to ADLS/Blob) and tight integration with Azure Stream Analytics and Functions.
  humanURL: https://azure.microsoft.com/en-us/products/event-hubs/
  tags:
  - Streaming
  - Azure
  - Kafka Compatible
  properties:
  - type: Documentation
    url: https://learn.microsoft.com/en-us/azure/event-hubs/
- aid: streaming:confluent-cloud
  name: Confluent Cloud
  description: Managed Kafka by the original Kafka authors. Cluster, topic, connector, KSQL, Schema Registry, Stream Governance,
    and Flink offerings exposed via a Confluent Cloud REST API and Terraform provider.
  humanURL: https://www.confluent.io/confluent-cloud/
  tags:
  - Streaming
  - Kafka
  - Managed
  properties:
  - type: Documentation
    url: https://docs.confluent.io/cloud/current/overview.html
  - type: Topic
    url: https://github.com/api-evangelist/confluent-the-data-streaming-platform
- aid: streaming:streamnative
  name: StreamNative
  description: Managed Apache Pulsar as a service from Pulsar's original contributors, with multi-cloud clusters, Functions,
    sources/sinks, and a control-plane REST API.
  humanURL: https://streamnative.io
  tags:
  - Streaming
  - Pulsar
  - Managed
  properties:
  - type: Documentation
    url: https://docs.streamnative.io
- aid: streaming:server-sent-events
  name: Server-Sent Events (SSE)
  description: One-directional HTTP-based streaming from server to client using the `text/event-stream` media type. Defined
    by the HTML Living Standard EventSource API; widely used for LLM token streams, dashboards, and live feeds where bidirectionality
    is not required.
  humanURL: https://html.spec.whatwg.org/multipage/server-sent-events.html
  tags:
  - Streaming
  - HTTP
  - Protocol
  - Standard
  properties:
  - type: Documentation
    url: https://html.spec.whatwg.org/multipage/server-sent-events.html
  - type: Documentation
    url: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
- aid: streaming:websocket
  name: WebSocket
  description: Full-duplex, bidirectional streaming protocol over a single TCP connection, upgraded from HTTP. RFC 6455. Foundation
    for chat, collaborative apps, market data, and real-time control planes.
  humanURL: https://datatracker.ietf.org/doc/html/rfc6455
  tags:
  - Streaming
  - Protocol
  - Standard
  - IETF
  properties:
  - type: Documentation
    url: https://datatracker.ietf.org/doc/html/rfc6455
  - type: Documentation
    url: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API
- aid: streaming:grpc-streaming
  name: gRPC Streaming
  description: 'gRPC defines four RPC styles, three of which are streaming: server streaming, client streaming, and bidirectional
    streaming, all multiplexed over HTTP/2. The default streaming surface for service-to-service systems and Kubernetes-native
    APIs.'
  humanURL: https://grpc.io
  tags:
  - Streaming
  - Protocol
  - HTTP2
  - CNCF
  properties:
  - type: Documentation
    url: https://grpc.io/docs/what-is-grpc/core-concepts/
  - type: GitHubRepository
    url: https://github.com/grpc/grpc
- aid: streaming:graphql-subscriptions
  name: GraphQL Subscriptions
  description: The GraphQL operation type for receiving a stream of updates over a long-lived transport (typically WebSocket
    via the graphql-ws or graphql-transport-ws sub-protocols, or SSE). Used to push schema- defined deltas to clients.
  humanURL: https://spec.graphql.org/draft/#sec-Subscription
  tags:
  - Streaming
  - GraphQL
  - Standard
  properties:
  - type: Documentation
    url: https://spec.graphql.org/draft/#sec-Subscription
  - type: Documentation
    url: https://github.com/enisdenjo/graphql-ws
  - url: graphql/streaming-graphql.md
    type: GraphQL
- aid: streaming:kafka-connect
  name: Kafka Connect
  description: Framework and runtime for source/sink connectors that move data into and out of Kafka. Distributed mode runs
    a REST-controlled cluster of workers managing connector and task lifecycle.
  humanURL: https://kafka.apache.org/documentation/#connect
  tags:
  - Streaming
  - Connectors
  - Kafka
  - Open Source
  properties:
  - type: Documentation
    url: https://kafka.apache.org/documentation/#connect
  - type: Topic
    url: https://github.com/api-evangelist/kafka-connect
- aid: streaming:debezium
  name: Debezium
  description: Change-data-capture (CDC) platform that streams row-level database changes (Postgres, MySQL, MongoDB, SQL Server,
    Oracle, Cassandra) as Kafka records using each database's native replication log.
  humanURL: https://debezium.io
  tags:
  - Streaming
  - Change Data Capture
  - Open Source
  - Red Hat
  properties:
  - type: Documentation
    url: https://debezium.io/documentation/
  - type: GitHubRepository
    url: https://github.com/debezium/debezium
- aid: streaming:apache-flink
  name: Apache Flink
  description: Distributed, stateful stream-processing engine with event-time semantics, windowing, watermarks, and exactly-once
    state. SQL, DataStream, and Table APIs; reference engine for sub-second latency analytics on streams.
  humanURL: https://flink.apache.org
  tags:
  - Streaming
  - Stream Processing
  - Open Source
  - Apache Software Foundation
  properties:
  - type: Documentation
    url: https://nightlies.apache.org/flink/flink-docs-stable/
  - type: GitHubRepository
    url: https://github.com/apache/flink
  - type: Topic
    url: https://github.com/api-evangelist/apache-flink
- aid: streaming:spark-structured-streaming
  name: Spark Structured Streaming
  description: Stream-processing API built on Spark SQL using a micro-batch (and experimental continuous) execution model.
    Treats a stream as an unbounded table.
  humanURL: https://spark.apache.org/streaming/
  tags:
  - Streaming
  - Stream Processing
  - Open Source
  - Apache Software Foundation
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
- aid: streaming:materialize
  name: Materialize
  description: Operational data warehouse and streaming SQL database built on Differential Dataflow. Maintains incrementally
    updated materialized views over streaming sources with millisecond freshness.
  humanURL: https://materialize.com
  tags:
  - Streaming
  - Streaming SQL
  - Database
  properties:
  - type: Documentation
    url: https://materialize.com/docs/
  - type: Topic
    url: https://github.com/api-evangelist/materialize
- aid: streaming:tinybird
  name: Tinybird
  description: Real-time analytics platform built on ClickHouse; ingests streams via HTTP, Kafka, or CDC, exposes SQL pipes
    as parameterized HTTP API endpoints with auth tokens.
  humanURL: https://www.tinybird.co
  tags:
  - Streaming
  - Real Time Analytics
  - ClickHouse
  properties:
  - type: Documentation
    url: https://www.tinybird.co/docs
- aid: streaming:bytewax
  name: Bytewax
  description: Open-source Python-native stream-processing framework built on Timely Dataflow; targets data scientists and
    Python teams building real-time ML and data pipelines.
  humanURL: https://bytewax.io
  tags:
  - Streaming
  - Stream Processing
  - Python
  - Open Source
  properties:
  - type: Documentation
    url: https://docs.bytewax.io
  - type: GitHubRepository
    url: https://github.com/bytewax/bytewax
- aid: streaming:apache-beam
  name: Apache Beam
  description: Unified batch and streaming programming model. Beam pipelines run on multiple runners (Dataflow, Flink, Spark,
    Samza), defining the canonical event-time / watermark / window / trigger semantics for stream processing.
  humanURL: https://beam.apache.org
  tags:
  - Streaming
  - Stream Processing
  - Open Source
  - Apache Software Foundation
  properties:
  - type: Documentation
    url: https://beam.apache.org/documentation/
  - type: GitHubRepository
    url: https://github.com/apache/beam
common:
- type: JSONSchema
  url: json-schema/streaming-stream-schema.json
- type: JSONSchema
  url: json-schema/streaming-stream-record-schema.json
- type: JSONSchema
  url: json-schema/streaming-stream-platform-schema.json
- type: JSONLD
  url: json-ld/streaming-context.jsonld
- type: Vocabulary
  url: vocabulary/streaming-vocabulary.yml
- type: Examples
  url: examples/streaming-stream-example.json
- type: Examples
  url: examples/streaming-stream-record-example.json
- type: Examples
  url: examples/streaming-stream-platform-example.json
include: []
maintainers:
- FN: Kin Lane
  email: info@apievangelist.com
  X-twitter: apievangelist