Apache Spark website screenshot

Apache Spark

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general execution graphs. Spark offers a comprehensive suite of APIs for batch processing, SQL queries, streaming analytics, machine learning, and graph computation, governed by the Apache Software Foundation.

Apache Spark publishes 1 API on the APIs.io network: REST API. Tagged areas include Analytics, Big Data, Distributed Computing, Machine Learning, and Open Source.

Apache Spark’s developer surface includes developer portal, documentation, getting-started guide, engineering blog, support, Stack Overflow tag, and 8 more developer resources.

🌐 Visit website 📡 Source on GitHub
37.1/100 thin ▬ flat Agent 39/100 agent aware Full breakdown ↓
scored 2026-07-21 · rubric v0.4
5 APIs 7 Features
AnalyticsBig DataDistributed ComputingMachine LearningOpen SourceStreaming

API Rating

API Evangelist API Evangelist Rating How this is scored →
scored 2026-07-21 · rubric v0.4
Composite quality — 37.1/100 · thin
Contract Quality 8.0 / 25
Developer Ergonomics 8.3 / 20
Commercial Clarity 10.0 / 20
Operational Transparency 4.1 / 13
Governance 0.0 / 12
Discoverability 6.8 / 10
Agent readiness — 39/100 · agent aware
Machine-Readable Contract 18 / 18
Agentic Access Contract 15 / 15
MCP Server 0 / 12
Machine-Readable Auth 0 / 10
Idempotency 0 / 9
Stable Error Semantics 0 / 8
Request/Response Examples 0 / 7
Rate-Limit Signaling 7 / 7
Typed Event Surface 0 / 6
Agent Skills 0 / 5
Well-Known Catalog 0 / 4
Consent & Bot Identity 0 / 3
Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. This rating is computed from github.com/api-evangelist/apache-spark: open an issue to ask a question, or submit a pull request to add artifacts. Want it done for you? Prioritized profiling — $2,500 →

APIs

Apache Spark REST API

REST API for monitoring Spark applications, accessing cluster information, and managing Spark jobs through the Spark UI backend. Exposes endpoints for applications, jobs, stages...

Apache Spark SQL API

Spark module for structured data processing with DataFrame and Dataset APIs. Provides a SQL interface and supports various data sources including Parquet, ORC, JSON, CSV, JDBC, ...

Apache Spark Streaming API

Scalable, high-throughput, fault-tolerant stream processing of live data streams. Supports Structured Streaming (the newer DStream-based API) with exactly-once semantics, contin...

Apache Spark MLlib API

Spark's scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dime...

Apache Spark GraphX API

Spark API for graphs and graph-parallel computation with a collection of graph algorithms and builders, including PageRank, Connected Components, Triangle Counting, and shortest...

Collections

Apache Spark Monitoring REST API

OPEN

Pricing Plans

Apache Spark Plans Pricing

3 plans

PLANS

Rate Limits

Apache Spark Rate Limits

5 limits

RATE LIMITS

FinOps

Apache Spark Finops

FINOPS

Features

Unified Analytics Engine

Single engine for batch, streaming, SQL, ML, and graph processing workloads.

Lazy Evaluation and DAG Execution

Optimized execution plans with Catalyst optimizer and DAG scheduling.

In-Memory Processing

Up to 100x faster than Hadoop MapReduce for iterative algorithms via in-memory caching.

Structured Streaming

Unified streaming and batch processing with exactly-once semantics and Kafka integration.

Multi-Language Support

High-level APIs in Scala, Java, Python (PySpark), and R (SparkR).

Delta Lake Integration

ACID transactions, schema evolution, and time travel for data lakes.

Kubernetes Native

Native Kubernetes scheduling for cloud-native deployment of Spark workloads.

Use Cases

Large-Scale ETL

Extract, transform, and load petabytes of data across distributed clusters.

Real-Time Analytics

Streaming analytics on live event data with sub-second latency.

Machine Learning Pipelines

Distributed ML training and feature engineering at scale with MLlib.

Data Lake Processing

Query and transform data stored in cloud object stores and HDFS.

Interactive SQL Analytics

Interactive SQL queries on structured and semi-structured data at scale.

Integrations

Apache Hadoop

HDFS storage, YARN cluster manager, and Hadoop ecosystem integration.

Apache Kafka

Structured Streaming source and sink for real-time event processing.

Delta Lake

Open-source storage layer with ACID transactions for data lakes.

Apache Iceberg

Open table format for huge analytic datasets on cloud storage.

Apache Hive

Hive metastore integration for table catalog and metadata management.

Kubernetes

Native Kubernetes scheduling for cloud-native Spark deployments.

Apache Airflow

Workflow orchestration for scheduling and managing Spark jobs.

Resources

Get Started 2

Portal, sign-up, and the first successful call

Portal
Portal
 GettingStarted
GettingStarted

Documentation 1

Reference material describing how the API behaves

Documentation
Documentation

Agent Surfaces 1

MCP servers, agent skills, and machine-readable catalogs

AgenticAccess
AgenticAccess

Build 3

SDKs, sample code, and the tooling you integrate with

GitHubRepository
GitHubRepository
 PySpark (Python)
SDKs
 Maven (Scala/Java)
SDKs

Access & Security 2

Authentication, authorization, and security posture

VulnerabilityDisclosure
VulnerabilityDisclosure
 DomainSecurity
DomainSecurity

Operate 2

Status, limits, changes, and where to get help

Support
Support
 StackOverflow
StackOverflow

Commercial 1

Pricing, plans, and the legal terms of use

TermsOfService
TermsOfService

Company 2

The organization behind the API

LinkedIn
LinkedIn
 Blog
Blog

Source (apis.yml)

apis.yml Raw ↑ 
opencollection: 1.0.0
info:
  name: Apache Spark Monitoring REST API
  version: '1.0'
items:
- info:
    name: List applications
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications
  docs: List applications
- info:
    name: List jobs for an application
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/jobs
    params:
    - name: app-id
      value: ''
      type: path
  docs: List jobs for an application
- info:
    name: Get a job
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/jobs/:job-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: job-id
      value: ''
      type: path
  docs: Get a job
- info:
    name: List stages
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages
    params:
    - name: app-id
      value: ''
      type: path
  docs: List stages
- info:
    name: Get stage
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
  docs: Get stage
- info:
    name: Get stage attempt
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id/:stage-attempt-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
    - name: stage-attempt-id
      value: ''
      type: path
  docs: Get stage attempt
- info:
    name: Stage attempt task summary
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id/:stage-attempt-id/taskSummary
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
    - name: stage-attempt-id
      value: ''
      type: path
  docs: Stage attempt task summary
- info:
    name: Stage attempt task list
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id/:stage-attempt-id/taskList
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
    - name: stage-attempt-id
      value: ''
      type: path
  docs: Stage attempt task list
- info:
    name: List active executors
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/executors
    params:
    - name: app-id
      value: ''
      type: path
  docs: List active executors
- info:
    name: Executor thread dump
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/executors/:executor-id/threads
    params:
    - name: app-id
      value: ''
      type: path
    - name: executor-id
      value: ''
      type: path
  docs: Executor thread dump
- info:
    name: List all executors (active and dead)
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/allexecutors
    params:
    - name: app-id
      value: ''
      type: path
  docs: List all executors (active and dead)
- info:
    name: List persisted RDDs
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/storage/rdd
    params:
    - name: app-id
      value: ''
      type: path
  docs: List persisted RDDs
- info:
    name: Get RDD details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/storage/rdd/:rdd-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: rdd-id
      value: ''
      type: path
  docs: Get RDD details
- info:
    name: Download event logs
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:base-app-id/logs
    params:
    - name: base-app-id
      value: ''
      type: path
  docs: Download event logs
- info:
    name: Download event logs for an attempt
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:base-app-id/:attempt-id/logs
    params:
    - name: base-app-id
      value: ''
      type: path
    - name: attempt-id
      value: ''
      type: path
  docs: Download event logs for an attempt
- info:
    name: Streaming statistics
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/statistics
    params:
    - name: app-id
      value: ''
      type: path
  docs: Streaming statistics
- info:
    name: Streaming receivers
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/receivers
    params:
    - name: app-id
      value: ''
      type: path
  docs: Streaming receivers
- info:
    name: Streaming receiver details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/receivers/:stream-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: stream-id
      value: ''
      type: path
  docs: Streaming receiver details
- info:
    name: Streaming batches
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches
    params:
    - name: app-id
      value: ''
      type: path
  docs: Streaming batches
- info:
    name: Streaming batch details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches/:batch-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: batch-id
      value: ''
      type: path
  docs: Streaming batch details
- info:
    name: Streaming batch operations
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches/:batch-id/operations
    params:
    - name: app-id
      value: ''
      type: path
    - name: batch-id
      value: ''
      type: path
  docs: Streaming batch operations
- info:
    name: Streaming output operation details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches/:batch-id/operations/:outputOp-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: batch-id
      value: ''
      type: path
    - name: outputOp-id
      value: ''
      type: path
  docs: Streaming output operation details
- info:
    name: List SQL executions
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/sql
    params:
    - name: app-id
      value: ''
      type: path
  docs: List SQL executions
- info:
    name: Get SQL execution details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/sql/:execution-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: execution-id
      value: ''
      type: path
  docs: Get SQL execution details
- info:
    name: Application environment
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/environment
    params:
    - name: app-id
      value: ''
      type: path
  docs: Application environment
- info:
    name: Spark version
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/version
  docs: Spark version
bundled: true