Apache Spark

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general execution graphs. Spark offers a comprehensive suite of APIs for batch processing, SQL queries, streaming analytics, machine learning, and graph computation, governed by the Apache Software Foundation.

5 APIs 7 Features

AnalyticsBig DataDistributed ComputingMachine LearningOpen SourceStreaming

APIs

Apache Spark REST API

REST API for monitoring Spark applications, accessing cluster information, and managing Spark jobs through the Spark UI backend. Exposes endpoints for applications, jobs, stages...

Apache Spark SQL API

Spark module for structured data processing with DataFrame and Dataset APIs. Provides a SQL interface and supports various data sources including Parquet, ORC, JSON, CSV, JDBC, ...

Apache Spark Streaming API

Scalable, high-throughput, fault-tolerant stream processing of live data streams. Supports Structured Streaming (the newer DStream-based API) with exactly-once semantics, contin...

Apache Spark MLlib API

Spark's scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dime...

Apache Spark GraphX API

Spark API for graphs and graph-parallel computation with a collection of graph algorithms and builders, including PageRank, Connected Components, Triangle Counting, and shortest...

Collections

Apache Spark Monitoring REST API

OPEN

Pricing Plans

Apache Spark Plans Pricing

3 plans

PLANS

Rate Limits

Apache Spark Rate Limits

5 limits

RATE LIMITS

FinOps

Apache Spark Finops

FINOPS

Features

Unified Analytics Engine

Single engine for batch, streaming, SQL, ML, and graph processing workloads.

Lazy Evaluation and DAG Execution

Optimized execution plans with Catalyst optimizer and DAG scheduling.

In-Memory Processing

Up to 100x faster than Hadoop MapReduce for iterative algorithms via in-memory caching.

Structured Streaming

Unified streaming and batch processing with exactly-once semantics and Kafka integration.

Multi-Language Support

High-level APIs in Scala, Java, Python (PySpark), and R (SparkR).

Delta Lake Integration

ACID transactions, schema evolution, and time travel for data lakes.

Kubernetes Native

Native Kubernetes scheduling for cloud-native deployment of Spark workloads.

Use Cases

Large-Scale ETL

Extract, transform, and load petabytes of data across distributed clusters.

Real-Time Analytics

Streaming analytics on live event data with sub-second latency.

Machine Learning Pipelines

Distributed ML training and feature engineering at scale with MLlib.

Data Lake Processing

Query and transform data stored in cloud object stores and HDFS.

Interactive SQL Analytics

Interactive SQL queries on structured and semi-structured data at scale.

Integrations

Apache Hadoop

HDFS storage, YARN cluster manager, and Hadoop ecosystem integration.

Apache Kafka

Structured Streaming source and sink for real-time event processing.

Delta Lake

Open-source storage layer with ACID transactions for data lakes.

Apache Iceberg

Open table format for huge analytic datasets on cloud storage.

Apache Hive

Hive metastore integration for table catalog and metadata management.

Kubernetes

Native Kubernetes scheduling for cloud-native Spark deployments.

Apache Airflow

Workflow orchestration for scheduling and managing Spark jobs.

Resources

Maven (Scala/Java)

SDKs

Sources

opencollection: 1.0.0
info:
  name: Apache Spark Monitoring REST API
  version: '1.0'
items:
- info:
    name: List applications
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications
  docs: List applications
- info:
    name: List jobs for an application
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/jobs
    params:
    - name: app-id
      value: ''
      type: path
  docs: List jobs for an application
- info:
    name: Get a job
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/jobs/:job-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: job-id
      value: ''
      type: path
  docs: Get a job
- info:
    name: List stages
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages
    params:
    - name: app-id
      value: ''
      type: path
  docs: List stages
- info:
    name: Get stage
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
  docs: Get stage
- info:
    name: Get stage attempt
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id/:stage-attempt-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
    - name: stage-attempt-id
      value: ''
      type: path
  docs: Get stage attempt
- info:
    name: Stage attempt task summary
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id/:stage-attempt-id/taskSummary
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
    - name: stage-attempt-id
      value: ''
      type: path
  docs: Stage attempt task summary
- info:
    name: Stage attempt task list
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/stages/:stage-id/:stage-attempt-id/taskList
    params:
    - name: app-id
      value: ''
      type: path
    - name: stage-id
      value: ''
      type: path
    - name: stage-attempt-id
      value: ''
      type: path
  docs: Stage attempt task list
- info:
    name: List active executors
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/executors
    params:
    - name: app-id
      value: ''
      type: path
  docs: List active executors
- info:
    name: Executor thread dump
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/executors/:executor-id/threads
    params:
    - name: app-id
      value: ''
      type: path
    - name: executor-id
      value: ''
      type: path
  docs: Executor thread dump
- info:
    name: List all executors (active and dead)
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/allexecutors
    params:
    - name: app-id
      value: ''
      type: path
  docs: List all executors (active and dead)
- info:
    name: List persisted RDDs
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/storage/rdd
    params:
    - name: app-id
      value: ''
      type: path
  docs: List persisted RDDs
- info:
    name: Get RDD details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/storage/rdd/:rdd-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: rdd-id
      value: ''
      type: path
  docs: Get RDD details
- info:
    name: Download event logs
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:base-app-id/logs
    params:
    - name: base-app-id
      value: ''
      type: path
  docs: Download event logs
- info:
    name: Download event logs for an attempt
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:base-app-id/:attempt-id/logs
    params:
    - name: base-app-id
      value: ''
      type: path
    - name: attempt-id
      value: ''
      type: path
  docs: Download event logs for an attempt
- info:
    name: Streaming statistics
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/statistics
    params:
    - name: app-id
      value: ''
      type: path
  docs: Streaming statistics
- info:
    name: Streaming receivers
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/receivers
    params:
    - name: app-id
      value: ''
      type: path
  docs: Streaming receivers
- info:
    name: Streaming receiver details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/receivers/:stream-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: stream-id
      value: ''
      type: path
  docs: Streaming receiver details
- info:
    name: Streaming batches
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches
    params:
    - name: app-id
      value: ''
      type: path
  docs: Streaming batches
- info:
    name: Streaming batch details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches/:batch-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: batch-id
      value: ''
      type: path
  docs: Streaming batch details
- info:
    name: Streaming batch operations
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches/:batch-id/operations
    params:
    - name: app-id
      value: ''
      type: path
    - name: batch-id
      value: ''
      type: path
  docs: Streaming batch operations
- info:
    name: Streaming output operation details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/streaming/batches/:batch-id/operations/:outputOp-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: batch-id
      value: ''
      type: path
    - name: outputOp-id
      value: ''
      type: path
  docs: Streaming output operation details
- info:
    name: List SQL executions
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/sql
    params:
    - name: app-id
      value: ''
      type: path
  docs: List SQL executions
- info:
    name: Get SQL execution details
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/sql/:execution-id
    params:
    - name: app-id
      value: ''
      type: path
    - name: execution-id
      value: ''
      type: path
  docs: Get SQL execution details
- info:
    name: Application environment
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/applications/:app-id/environment
    params:
    - name: app-id
      value: ''
      type: path
  docs: Application environment
- info:
    name: Spark version
    type: http
  http:
    method: GET
    url: http://localhost:4040/api/v1/version
  docs: Spark version
bundled: true