Streaming
Streaming is a topic catalog of the protocols, platforms, and processing engines used to move and transform real-time, high-volume, often bidirectional data over the network. It indexes the canonical log-structured and broker systems (Apache Kafka, Apache Pulsar, Redpanda, NATS JetStream, AWS Kinesis, GCP Pub/Sub + Dataflow, Azure Event Hubs, Confluent Cloud, StreamNative), the over-the-wire streaming protocols exposed to API consumers (Server-Sent Events, WebSocket, gRPC streaming, GraphQL subscriptions), the change-data capture and connector frameworks that feed them (Kafka Connect, Debezium), and the stream-processing engines that consume them (Apache Flink, Spark Structured Streaming, Materialize, Tinybird, Bytewax, Apache Beam). This topic is distinguished from `events` and `async-apis`: streaming emphasizes real-time, high-throughput, partitioned, and often bidirectional pipes, rather than discrete event envelopes or static contract documents.
Resources
-
Apache Kafka
Distributed, partitioned, replicated log. The reference open-source streaming platform; durable, ordered topics with consumer groups, exactly -once semantics, and the de facto wire protocol for the streaming ecosystem. Native Kafka clients…
-
Apache Pulsar
Cloud-native, multi-tenant pub/sub and streaming platform with a tiered storage architecture (BookKeeper) that separates compute from storage, native geo-replication, and built-in Functions for lightweight stream processing.
-
Redpanda
Kafka-API-compatible streaming platform implemented in C++ with no ZooKeeper/JVM dependency. Single binary, thread-per-core architecture, Raft consensus; positioned as a drop-in for Kafka workloads.
-
NATS JetStream
Persistence layer for the NATS messaging system providing at-least-once and exactly-once streaming, key/value and object stores, and durable consumers — designed for edge, IoT, and microservice topologies.
-
Amazon Kinesis
AWS managed family for real-time streaming: Kinesis Data Streams (shards, partition keys, 24h–365d retention), Kinesis Data Firehose (delivery to S3/Redshift/OpenSearch), and Kinesis Video Streams for media. HTTP/2 SubscribeToShard for low…
-
Google Cloud Pub/Sub and Dataflow
GCP's managed messaging (Pub/Sub) and stream-processing (Dataflow, built on Apache Beam) stack. Pub/Sub provides at-least-once and exactly-once delivery with push/pull subscribers; Dataflow runs windowed, watermark-aware Beam pipelines.
-
Azure Event Hubs
Microsoft's managed big-data streaming platform; Kafka-protocol compatible, partitioned, with Capture (delivery to ADLS/Blob) and tight integration with Azure Stream Analytics and Functions.
-
Confluent Cloud
Managed Kafka by the original Kafka authors. Cluster, topic, connector, KSQL, Schema Registry, Stream Governance, and Flink offerings exposed via a Confluent Cloud REST API and Terraform provider.
-
StreamNative
Managed Apache Pulsar as a service from Pulsar's original contributors, with multi-cloud clusters, Functions, sources/sinks, and a control-plane REST API.
-
Server-Sent Events (SSE)
One-directional HTTP-based streaming from server to client using the `text/event-stream` media type. Defined by the HTML Living Standard EventSource API; widely used for LLM token streams, dashboards, and live feeds where bidirectionality…
-
WebSocket
Full-duplex, bidirectional streaming protocol over a single TCP connection, upgraded from HTTP. RFC 6455. Foundation for chat, collaborative apps, market data, and real-time control planes.
-
gRPC Streaming
gRPC defines four RPC styles, three of which are streaming: server streaming, client streaming, and bidirectional streaming, all multiplexed over HTTP/2. The default streaming surface for service-to-service systems and Kubernetes-native AP…
-
GraphQL Subscriptions
The GraphQL operation type for receiving a stream of updates over a long-lived transport (typically WebSocket via the graphql-ws or graphql-transport-ws sub-protocols, or SSE). Used to push schema- defined deltas to clients.
-
Kafka Connect
Framework and runtime for source/sink connectors that move data into and out of Kafka. Distributed mode runs a REST-controlled cluster of workers managing connector and task lifecycle.
-
Debezium
Change-data-capture (CDC) platform that streams row-level database changes (Postgres, MySQL, MongoDB, SQL Server, Oracle, Cassandra) as Kafka records using each database's native replication log.
-
Apache Flink
Distributed, stateful stream-processing engine with event-time semantics, windowing, watermarks, and exactly-once state. SQL, DataStream, and Table APIs; reference engine for sub-second latency analytics on streams.
-
Spark Structured Streaming
Stream-processing API built on Spark SQL using a micro-batch (and experimental continuous) execution model. Treats a stream as an unbounded table.
-
Materialize
Operational data warehouse and streaming SQL database built on Differential Dataflow. Maintains incrementally updated materialized views over streaming sources with millisecond freshness.
-
Tinybird
Real-time analytics platform built on ClickHouse; ingests streams via HTTP, Kafka, or CDC, exposes SQL pipes as parameterized HTTP API endpoints with auth tokens.
-
Bytewax
Open-source Python-native stream-processing framework built on Timely Dataflow; targets data scientists and Python teams building real-time ML and data pipelines.
-
Apache Beam
Unified batch and streaming programming model. Beam pipelines run on multiple runners (Dataflow, Flink, Spark, Samza), defining the canonical event-time / watermark / window / trigger semantics for stream processing.
Links
Providers working in Streaming
Providers whose own tags share at least two of this topic's tags, most shared first — the top 30 of 139.
| Provider | About | Rating | APIs |
|---|---|---|---|
| Redpanda | Redpanda is a Kafka API-compatible streaming data platform written in C++ with no JVM and no ZooKeeper, optimized for low latency and operational simplicity. The core broker (redpanda) is open source and available under the Business Source… | thin | 10 |
| Streamkap | Streamkap is a real-time streaming ETL and change data capture (CDC) platform built on Apache Kafka and Apache Flink. It streams data from operational databases (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle) to cloud warehouses, lakes, a… | thin | 1 |
| Nstream | Nstream is the company behind SwimOS, an open-source platform for building streaming data applications. Instead of polling databases, Nstream keeps application state continuously in memory as stateful "web agents" that ingest events from a… | emerging | 0 |
| Heroic Labs | Heroic Labs is the company behind Nakama, a leading open-source game backend server providing a comprehensive REST, WebSocket, and gRPC API for building scalable multiplayer and social games. The platform delivers essential backend service… | exemplar | 4 |
| Buf | Buf Technologies builds the modern toolchain for Protocol Buffers and gRPC: the buf CLI, the Buf Schema Registry (BSR), Protovalidate, Protobuf-ES and Protobuf-Py, and the Connect protocol, which is now a CNCF project. It replaces protoc-b… | strong | 2 |
| S2 Dev | S2 ("Stream Store") is the API for unlimited, durable, real-time streams. Where object storage deals with blobs, S2 provides append-able, ordered record streams that can be tailed in real time and replayed from any retained point. Core dat… | strong | 1 |
| Amazon MSK | Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data, with the infrastructure management handled by AWS. | strong | 2 |
| LaserData | LaserData is a hyper-efficient data streaming platform built in Rust for AI-native, real-time, and latency-sensitive workloads. Founded by the creators of Apache Iggy, LaserData packages the Iggy message-streaming engine — io_uring, thread… | strong | 3 |
| WarpStream | WarpStream is a diskless, Apache Kafka-compatible data streaming platform built directly on top of cloud object storage such as S3, GCP, and Azure. It eliminates the need for local disks, brokers to rebalance, and ZooKeeper, delivering Kaf… | developing | 1 |
| Macrometa | Macrometa is a global data network and edge computing platform. Its Global Data Network (GDN) provides a geo-distributed, serverless NoSQL database, pub/sub streams, complex event processing, and edge functions through a unified REST API p… | developing | 8 |
| Striim | Unified data integration and streaming platform offering change data capture (CDC), real-time streaming analytics, and data validation. Exposes a token-authenticated REST API (WActionStore queries, system health, Application Management) co… | developing | 6 |
| Neuphonic | Neuphonic is an ultra-low-latency voice AI platform specializing in real-time text-to-speech synthesis with sub-25ms latency, making it suitable for conversational AI and live applications. The platform provides both a cloud-hosted API wit… | developing | 1 |
| RisingWave | RisingWave is a distributed SQL streaming platform that continuously ingests event streams from Kafka, Kinesis, and other sources, transforms them using PostgreSQL-compatible SQL, and serves low-latency results through incrementally mainta… | developing | 6 |
| Ably | Ably is a realtime messaging platform offering pub/sub, presence, push notifications, chat, LiveSync, and integrations over WebSocket and HTTP. Ably publishes its OpenAPI specifications publicly via the ably/open-specs GitHub repository, w… | developing | 2 |
| Quix | Quix is a Python-native stream-processing platform for real-time data and ML. It pairs Quix Streams - an open source (Apache 2.0) Python library for building containerized stream-processing applications on Apache Kafka - with Quix Cloud, a… | developing | 1 |
| AISStream | AISStream.io is a free service that streams global real-time AIS (Automatic Identification System) vessel-tracking data over a WebSocket. Clients register for a free API key, open a secure WebSocket to "wss://stream.aisstream.io/v0/stream"… | thin | 1 |
| Estuary Flow | Estuary Flow is a real-time data movement and transformation platform combining streaming infrastructure, a runtime, and an open-source ecosystem of connectors. It supports change data capture (CDC), SaaS integration, database replication,… | thin | 4 |
| Bytewax | Bytewax is a Python-native distributed stream processing framework built on a Rust runtime. Developers define dataflows using the bytewax.dataflow API, composing operators (map, filter, reduce, joins, windowing) over connectors for Kafka,… | thin | 3 |
| Capital.com Public API | Capital.com is a European trading platform offering CFDs across shares, indices, commodities, forex, and cryptocurrencies. The Capital.com Public API provides direct access to the trading engine for automated strategies, giving programmati… | thin | 8 |
| Pulsoid | Pulsoid enables real-time heart rate data transmission from peripherals (BLE heart rate monitors, smartwatches, etc.) to clients. The Pulsoid API allows reading and writing real-time heart rate data, accessing statistics, and managing widg… | thin | 1 |
| Apache Samza | Apache Samza is a distributed stream processing framework that provides a simple API for building stateful stream processing applications. It integrates with Apache Kafka for messaging and supports both stream and batch processing. | thin | 1 |
| Concord Systems | Concord Systems was a Brooklyn, New York stream-processing company founded in December 2014 by Alexander Gallego and Emilio Del Tesoro, and backed by Bloomberg Beta. It built a high-performance distributed stream processing framework writt… | minimal | 0 |
| Confluent | Stream, connect, process, and govern your data with an all-in-one, real-time platform from the pioneer in data streaming. Build faster, scale smarter, and turn data chaos into instantly accessible and usable data products with the market l… | exemplar | 3 |
| 0xArchive | 0xArchive is a replayable market-data archive for two decentralised perpetuals venues, Hyperliquid and Lighter, delivered as one REST API, one WebSocket API that carries both live subscriptions and historical replay on a single connection,… | exemplar | 2 |
| Massive (formerly Polygon.io) | Polygon (Polygon.io, rebranded as Massive in early 2026) provides real-time and historical market data APIs across US stocks, options, indices, forex, cryptocurrencies, and futures. Coverage is delivered through REST endpoints and WebSocke… | exemplar | 7 |
| Confluent | the Data Streaming Platform | Confluent is a fully managed data streaming platform built by the original creators of Apache Kafka. It lets organizations stream, connect, process, and govern data in motion through a cloud-native service (Confluent Cloud) and the on-prem… | exemplar | 2 |
| Kraken | Kraken is one of the world's largest and longest-running cryptocurrency exchanges, founded in 2011 and headquartered in San Francisco. Kraken offers Spot and Futures trading, staking and yield through Kraken Earn, NFT marketplace access, O… | exemplar | 24 |
| Gemini Trust Company | Gemini Trust Company, LLC is a United States cryptocurrency exchange and custodian, founded in 2014 and headquartered in New York, regulated as a New York State Department of Financial Services trust company. It operates the Gemini Exchang… | exemplar | 2 |
| Postman | Postman is the world's leading API platform, used by 35+ million developers to design, build, test, document, mock, monitor, and govern APIs across the entire API lifecycle. The platform spans Collections, Workspaces, the API Client, Spec… | exemplar | 21 |
| Amazon API Gateway | Amazon API Gateway is a fully managed service that makes it easy to create, publish, maintain, monitor, and secure APIs at any scale. It acts as the front door for applications to access backend services, supporting REST APIs, HTTP APIs, a… | exemplar | 13 |