Apache Spark

aid: apache-spark
name: Apache Spark
description: Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java,
  Scala, Python, and R, and an optimized engine that supports general execution graphs. Spark offers a comprehensive suite
  of APIs for batch processing, SQL queries, streaming analytics, machine learning, and graph computation, governed by the
  Apache Software Foundation.
type: Index
deliveryModel:
  model: open-core
  license: Apache-2.0
  open_source: true
  commercial: true
  callable_host: false
  label: Open core · an OSS project plus a commercial hosted product
  confidence: high
  source:
  - license
  - openapi
  - pricing
  generated: '2026-08-28'
  method: derived
accessModel:
  pricing: freemium
  onboarding: unknown
  trial: false
  try_now: false
  public: false
  label: Freemium
  confidence: low
  source:
  - plans
  - security
  generated: '2026-11-06'
  method: derived
position: Consuming
access: 3rd-Party
image: https://spark.apache.org/images/spark-logo-trademark.png
tags:
- Analytics
- Big Data
- Distributed Computing
- Machine Learning
- Open Source
- Streaming
tags_raw:
- Analytics
- Big Data
- Distributed Computing
- Machine Learning
- Open Source
- Streaming
- Machine-Learning
- Open-Source
created: '2024-01-01'
modified: '2026-05-19'
url: https://raw.githubusercontent.com/api-evangelist/apache-spark/refs/heads/main/apis.yml
specificationVersion: '0.23'
apis:
- aid: apache-spark:apache-spark-sql-api
  name: Apache Spark SQL API
  description: Spark module for structured data processing with DataFrame and Dataset APIs. Provides a SQL interface and supports
    various data sources including Parquet, ORC, JSON, CSV, JDBC, Hive, and Delta Lake. The Spark SQL API supports Scala,
    Python, Java, and R bindings.
  humanURL: https://spark.apache.org/docs/latest/sql-programming-guide.html
  tags:
  - DataFrames
  - SQL
  - Structured Data
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/sql-programming-guide.html
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/index.html
    title: Scala API Reference
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/index.html
    title: Python API Reference
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/sql/package-summary.html
    title: Java API Reference
- aid: apache-spark:apache-spark-streaming-api
  name: Apache Spark Streaming API
  description: Scalable, high-throughput, fault-tolerant stream processing of live data streams. Supports Structured Streaming
    (the newer DStream-based API) with exactly-once semantics, continuous processing mode, and integration with Kafka, Kinesis,
    HDFS, and other sources.
  humanURL: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
  tags:
  - Data Processing
  - Real-Time
  - Streaming
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/streaming/index.html
    title: Scala Streaming API
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.streaming/index.html
    title: Python Streaming API
- aid: apache-spark:apache-spark-mllib-api
  name: Apache Spark MLlib API
  description: Spark's scalable machine learning library consisting of common learning algorithms and utilities, including
    classification, regression, clustering, collaborative filtering, dimensionality reduction, and feature engineering. Supports
    pipeline-based ML workflows through the spark.ml package.
  humanURL: https://spark.apache.org/docs/latest/ml-guide.html
  tags:
  - Algorithms
  - Data Science
  - Machine Learning
  tags_raw:
  - Algorithms
  - Data Science
  - Machine Learning
  - ML
  - Machine-Learning
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/ml-guide.html
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/index.html
    title: Scala MLlib API
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html
    title: Python MLlib API
- aid: apache-spark:apache-spark-graphx-api
  name: Apache Spark GraphX API
  description: Spark API for graphs and graph-parallel computation with a collection of graph algorithms and builders, including
    PageRank, Connected Components, Triangle Counting, and shortest paths.
  humanURL: https://spark.apache.org/docs/latest/graphx-programming-guide.html
  tags:
  - Analytics
  - Graph Processing
  - Graphs
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/graphx-programming-guide.html
  - type: SDKs
    url: https://spark.apache.org/docs/latest/api/scala/org/apache/spark/graphx/index.html
    title: Scala GraphX API
- aid: apache-spark:apache-spark-applications-api
  name: Apache Spark Applications API
  description: The Applications API from Apache Spark — 25 operation(s) for applications.
  humanURL: https://spark.apache.org/docs/latest/monitoring.html#rest-api
  tags:
  - Application
  tags_raw:
  - Applications
  properties:
  - type: OpenAPI
    url: openapi/apache-spark-applications-api-openapi.yml
  - type: Documentation
    url: https://spark.apache.org/docs/latest/monitoring.html#rest-api
- aid: apache-spark:apache-spark-version-api
  name: Apache Spark Version API
  description: The Version API from Apache Spark — 1 operation(s) for version.
  humanURL: https://spark.apache.org/docs/latest/monitoring.html#rest-api
  tags:
  - Version
  properties:
  - type: OpenAPI
    url: openapi/apache-spark-version-api-openapi.yml
  - type: Documentation
    url: https://spark.apache.org/docs/latest/monitoring.html#rest-api
common:
- type: Plans
  url: plans/apache-spark-plans-pricing.yml
- type: RateLimits
  url: rate-limits/apache-spark-rate-limits.yml
- type: FinOps
  url: finops/apache-spark-finops.yml
- type: Website
  url: https://www.apache.org/
- type: IssueTracker
  url: https://github.com/apache/spark/issues
- type: SecurityPolicy
  url: https://github.com/apache/spark/blob/master/SECURITY.md
- type: CodeOfConduct
  url: https://github.com/apache/.github/blob/main/.github/CODE_OF_CONDUCT.md
- type: ContributionGuide
  url: https://github.com/apache/spark/blob/master/CONTRIBUTING.md
- type: License
  name: Apache-2.0
  url: https://github.com/apache/spark/blob/master/LICENSE
- type: AgenticAccess
  url: agentic-access/apache-spark-agentic-access.yml
- type: VulnerabilityDisclosure
  url: security/apache-spark-vulnerability-disclosure.yml
- type: DomainSecurity
  url: security/apache-spark-domain-security.yml
- type: LinkedIn
  url: https://www.linkedin.com/company/apachespark
- type: GitHubRepository
  url: https://github.com/apache/spark
- type: Portal
  url: https://spark.apache.org/
- type: Documentation
  url: https://spark.apache.org/docs/latest/
- type: GettingStarted
  url: https://spark.apache.org/docs/latest/quick-start.html
- type: Blog
  url: https://spark.apache.org/news/
- type: Support
  url: https://spark.apache.org/community.html
- type: TermsOfService
  url: https://www.apache.org/licenses/LICENSE-2.0
- type: StackOverflow
  url: https://stackoverflow.com/questions/tagged/apache-spark
- type: SDKs
  url: https://pypi.org/project/pyspark/
  title: PySpark (Python)
- type: SDKs
  url: https://search.maven.org/search?q=g:org.apache.spark
  title: Maven (Scala/Java)
- type: Features
  data:
  - name: Unified Analytics Engine
    description: Single engine for batch, streaming, SQL, ML, and graph processing workloads.
  - name: Lazy Evaluation and DAG Execution
    description: Optimized execution plans with Catalyst optimizer and DAG scheduling.
  - name: In-Memory Processing
    description: Up to 100x faster than Hadoop MapReduce for iterative algorithms via in-memory caching.
  - name: Structured Streaming
    description: Unified streaming and batch processing with exactly-once semantics and Kafka integration.
  - name: Multi-Language Support
    description: High-level APIs in Scala, Java, Python (PySpark), and R (SparkR).
  - name: Delta Lake Integration
    description: ACID transactions, schema evolution, and time travel for data lakes.
  - name: Kubernetes Native
    description: Native Kubernetes scheduling for cloud-native deployment of Spark workloads.
- type: UseCases
  data:
  - name: Large-Scale ETL
    description: Extract, transform, and load petabytes of data across distributed clusters.
  - name: Real-Time Analytics
    description: Streaming analytics on live event data with sub-second latency.
  - name: Machine Learning Pipelines
    description: Distributed ML training and feature engineering at scale with MLlib.
  - name: Data Lake Processing
    description: Query and transform data stored in cloud object stores and HDFS.
  - name: Interactive SQL Analytics
    description: Interactive SQL queries on structured and semi-structured data at scale.
- type: Integrations
  data:
  - name: Apache Hadoop
    description: HDFS storage, YARN cluster manager, and Hadoop ecosystem integration.
  - name: Apache Kafka
    description: Structured Streaming source and sink for real-time event processing.
  - name: Delta Lake
    description: Open-source storage layer with ACID transactions for data lakes.
  - name: Apache Iceberg
    description: Open table format for huge analytic datasets on cloud storage.
  - name: Apache Hive
    description: Hive metastore integration for table catalog and metadata management.
  - name: Kubernetes
    description: Native Kubernetes scheduling for cloud-native Spark deployments.
  - name: Apache Airflow
    description: Workflow orchestration for scheduling and managing Spark jobs.
maintainers:
- FN: Kin Lane
  email: info@apievangelist.com