Apache PySpark

Python API for Apache Spark - A unified analytics engine for large-scale data processing supporting batch processing, streaming, machine learning, and graph computing.

Apache PySpark is listed with 5 APIs on the APIs.io network. Tagged areas include Big Data, Data Processing, Distributed Computing, Machine Learning, and Python.

Apache PySpark’s developer surface includes getting-started guide, release notes, and 13 more developer resources.

🌐 Visit website 📡 Profile data on GitHub

17.3/100 emerging ▬ flat Agent 1/100 human only Front door AI 4/6 moderate Heartbeat 70/100 slowing Full breakdown ↓
scored 2026-11-13 · rubric v0.28.0 · Show this score on your site →

AccessFreemium

5 APIs

Big Data Data Processing Distributed Computing Machine Learning Python Streaming

Kin Score

Kin Score How this is scored →
scored 2026-11-13 · rubric v0.28.0

Composite quality — 17.3/100 · emerging

Contract Quality 0.0 / 21

Developer Ergonomics 5.7 / 17

Access Clarity 1.1 / 17

Operational Transparency 3.9 / 11

Contract Governance 0.0 / 10

Discoverability 4.6 / 9

Regulatory Posture 1.6 / 15

Agent readiness — 1/100 · human only

Machine-Readable Contract 0 / 18

Agentic Access Contract 0 / 10

Documented Reversibility 0 / 6

MCP Server 0 / 12

Machine-Readable Auth 0 / 10

Idempotency 0 / 9

Stable Error Semantics 0 / 8

Request/Response Examples 0 / 7

Rate-Limit Signaling 7 / 7

Typed Event Surface 0 / 6

Agent Skills 0 / 5

Well-Known Catalog 0 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Delegated User Identity 0 / 6

Protected Resource Metadata 0 / 5

Registration Without a Human 0 / 6

Agentic Commerce Well-Known Document 0 / 5

Pagination 5 / 5

Distinct Operations 3 / 3

MCP Tool Quality 6 / 6

Regulatory Posture applies to this provider. Its tags matched the Horizontal (data, software, accessibility, platform) regime, so Regulatory Posture carries 15 points of the composite. If this regime is wrong for your business, say so on your provider repo — the applicability map is public and we will correct it.

Create-or-Update Ergonomics could not be measured. We hold no machine-readable contract for this provider to read, so there is nothing to measure a write surface against. Excluded rather than scored zero: never-measured and measured-empty are different facts. Publishing an OpenAPI is what makes this facet — and several others — scorable at all.

The six quality facets above are damped to 85 points between them, because the conditional facet above carries the other 15. That is why each facet's contribution is shown against a damped maximum: raising a quality facet moves the composite by 85% of its nominal weight, not 100%. The full arithmetic is at apis.io/rating/.

Show this score on your site.

It updates itself when the score changes. Paste this where you link to your API docs:

<a href="https://apis.io/providers/pyspark/"><img src="https://apis.io/badge/pyspark/card.svg" alt="Apache PySpark on APIs.io — Kin Score by API Evangelist"></a>

Other styles, a README pill and a shields.io endpoint are on the badge page.

Raise your score.

Agent readiness — checks not met, largest first (exact, from this profile):

Machine-Readable Contract +18 pts — api_specs includes an OpenAPI document Ask about this →
MCP Server +12 pts — deployment.mode + probe verdict on the mcp/ manifest, not the pointer; an unprobed slug falls back to authorship Ask about this →
Agentic Access Contract +10 pts — agentic_access artifact present (x-agentic-access), graded by author Ask about this →
Machine-Readable Auth +10 pts — a served openid-configuration/oauth-authorization-server carrying `issuer` (served), else the securitySchemes class in the provider OpenAPIs — bound (mutualTLS, DPoP, Signature) > negotiable (oauth2 authorizationCode/clientCredentials, openIdConnect) > bearer (http bearer, apiKey) Ask about this →
Idempotency +9 pts — OpenAPI mutating operations declare an idempotency key — a header, a request-body property, or a named de-duplication token whose own description says so (verified); else common[].type matches /idempoten/i (documented) Ask about this →
Stable Error Semantics +8 pts — OpenAPI $refs a shared error schema across 4xx/5xx responses (verified), else common[].type includes ErrorCatalog / ErrorCodes / DeclineCodes Ask about this →

12 more unmet checks in the Agent Readiness table above.

Kin Score — facets with the most composite points still available (weight × headroom):

Contract Quality scores 0.0 — up to +25.0 composite points Ask about this →
Access Clarity scores 6.4 — up to +18.7 composite points Ask about this →
Developer Ergonomics scores 33.3 — up to +13.3 composite points Ask about this →

Each check links to what satisfies it. Publish the artifact, or tell us where it already lives, and the score moves on the next scoring run.

Not sure how this applies to your stack? Ask about any one item. Kin Lane reads every question, looks at your profile and your situation, and replies with what a fix would take — scoped to that one area, priced up front, yours whether you ship it or not. Ask a question →

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from the provider profile at artifacts.apis.io/pyspark/apis.yml: open an issue on github.com/api-evangelist/pyspark to ask a question or point us at an artifact you publish. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

APIs 5

Individual APIs this provider publishes, each with its own machine-readable definition.

PySpark Core API

Core Spark functionality including RDDs, SparkContext, and basic operations.

PySpark SQL

Structured data processing with DataFrame and SQL operations.

PySpark Streaming

Real-time stream processing capabilities using DStreams and Structured Streaming.

PySpark MLlib

Machine learning library with scalable algorithms for classification, regression, clustering, and more.

PySpark ML (DataFrame-based)

DataFrame-based machine learning API with pipelines and feature transformers.

Apache PySpark API pricing

Not yet checked against Apache PySpark's own pricing page. Compiled by API Evangelist on 2026-05-04; confirm before you rely on a figure. A free plan or tier is listed.

Plan	Price
Free freemium	Free
Professional paid	Free
Enterprise enterprise	—

Plan details, limits and sources →

Rate limits

Free Tier Default: 10 requests per minute
Free Tier Monthly Quota: 1000 requests per month
Professional Tier Default: 100 requests per minute

Compiled by API Evangelist, not yet checked against Apache PySpark's documentation. All 5 limits →

Pricing Plans 1

Published pricing tiers and plan structures.

Apache PySpark Plans Pricing

3 plans

PLANS

Rate Limits 1

Documented rate limits and quota policies.

Apache PySpark Rate Limits

5 limits

RATE LIMITS

FinOps 1

Cost, billing, and metering signals for API financial operations.

Apache PySpark Finops

FINOPS

Security Posture 2

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Apache PySpark Domain Security

TLSv1.3 · HSTS · DMARC

SECURITY

Apache PySpark Vulnerability Disclosure

security.txt · contact published

SECURITY

Resources

Get Started 2

Portal, sign-up, and the first successful call

GettingStarted

GettingStarted

GettingStarted

GettingStarted

Build 1

SDKs, sample code, and the tooling you integrate with

GitHubOrganization

GitHubOrganization

Access & Security 3

Authentication, authorization, and security posture

VulnerabilityDisclosure

VulnerabilityDisclosure

DomainSecurity

DomainSecurity

Security

Security

Operate 4

Status, limits, changes, and where to get help

RateLimits

Community

IssueTracker

ReleaseNotes

Commercial 2

Pricing, plans, and the legal terms of use

Plans

Plans

FinOps

FinOps

Company 2

The organization behind the API

LinkedIn

Website

Website

Other 1

Properties that don't map to a standard resource type

Downloads

Downloads

Source (apis.yml)

aid: pyspark
name: Apache PySpark
description: Python API for Apache Spark - A unified analytics engine for large-scale data processing supporting batch processing,
  streaming, machine learning, and graph computing.
type: Index
deliveryModel:
  model: unknown
  open_source: false
  commercial: false
  callable_host: false
  label: Delivery model not determined — needs a product licence on record
  confidence: low
  source:
  - none
  generated: '2026-08-28'
  method: derived
accessModel:
  pricing: freemium
  onboarding: unknown
  trial: false
  try_now: false
  public: false
  label: Freemium
  confidence: low
  source:
  - plans
  - security
  generated: '2026-11-13'
  method: derived
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/pyspark.png
tags:
- Big Data
- Data Processing
- Distributed Computing
- Machine Learning
- Python
- Streaming
tags_raw:
- Big Data
- Data Processing
- Distributed Computing
- Machine Learning
- Python
- Streaming
- Machine-Learning
url: https://raw.githubusercontent.com/api-evangelist/pyspark/refs/heads/main/apis.yml
created: '2024-01-01'
modified: '2026-04-28'
specificationVersion: '0.23'
apis:
- aid: pyspark:pyspark-core-api
  name: PySpark Core API
  description: Core Spark functionality including RDDs, SparkContext, and basic operations.
  humanURL: https://spark.apache.org/docs/latest/api/python/reference/pyspark.html
  tags:
  - RDD
  - Spark Context
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.html
  - type: APIReference
    url: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.html
- aid: pyspark:pyspark-sql
  name: PySpark SQL
  description: Structured data processing with DataFrame and SQL operations.
  humanURL: https://spark.apache.org/docs/latest/sql-programming-guide.html
  tags:
  - DataFrames
  - SQL
  tags_raw:
  - DataFrame
  - SQL
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/index.html
  - type: GettingStarted
    url: https://spark.apache.org/docs/latest/sql-getting-started.html
- aid: pyspark:pyspark-streaming
  name: PySpark Streaming
  description: Real-time stream processing capabilities using DStreams and Structured Streaming.
  humanURL: https://spark.apache.org/docs/latest/streaming-programming-guide.html
  tags:
  - Streaming
  - Real-Time
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.streaming/index.html
  - type: ProgrammingGuide
    url: https://spark.apache.org/docs/latest/streaming-programming-guide.html
- aid: pyspark:pyspark-mllib
  name: PySpark MLlib
  description: Machine learning library with scalable algorithms for classification, regression, clustering, and more.
  humanURL: https://spark.apache.org/docs/latest/ml-guide.html
  tags:
  - Machine Learning
  - MLlib
  tags_raw:
  - Machine Learning
  - MLlib
  - Machine-Learning
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html
  - type: MLGuide
    url: https://spark.apache.org/docs/latest/ml-guide.html
- aid: pyspark:pyspark-ml
  name: PySpark ML (DataFrame-based)
  description: DataFrame-based machine learning API with pipelines and feature transformers.
  humanURL: https://spark.apache.org/docs/latest/ml-pipeline.html
  tags:
  - Machine Learning
  - Pipelines
  tags_raw:
  - Machine Learning
  - Pipeline
  - Machine-Learning
  properties:
  - type: Documentation
    url: https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html
  - type: PipelineGuide
    url: https://spark.apache.org/docs/latest/ml-pipeline.html
common:
- type: Plans
  url: plans/pyspark-plans-pricing.yml
- type: RateLimits
  url: rate-limits/pyspark-rate-limits.yml
- type: FinOps
  url: finops/pyspark-finops.yml
- type: VulnerabilityDisclosure
  url: security/pyspark-vulnerability-disclosure.yml
- type: DomainSecurity
  url: security/pyspark-domain-security.yml
- type: LinkedIn
  url: https://www.linkedin.com/company/apachespark
- type: Website
  url: https://spark.apache.org/
- type: GitHubOrganization
  url: https://github.com/apache/spark
- type: GettingStarted
  url: https://spark.apache.org/docs/latest/api/python/getting_started/install.html
- type: GettingStarted
  url: https://spark.apache.org/docs/latest/quick-start.html
- type: Downloads
  url: https://spark.apache.org/downloads.html
- type: Community
  url: https://spark.apache.org/community.html
- type: IssueTracker
  url: https://issues.apache.org/jira/projects/SPARK
- type: ReleaseNotes
  url: https://spark.apache.org/releases/
- type: Security
  url: https://spark.apache.org/security.html
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com

Apache PySpark API — frequently asked questions

Does Apache PySpark have an API?

Yes. APIs.io lists 5 APIs from Apache PySpark, including PySpark Core API, PySpark SQL and PySpark Streaming, and 2 more.

How much does the Apache PySpark API cost?

APIs.io lists 3 pricing plans for Apache PySpark, with each plan's price, limits and inclusions under Pricing on this page.

Is the Apache PySpark API ready for AI agents?

Apache PySpark has a Kin Score of 17.3 out of 100 (emerging) and an Agent Readiness score of 0.6 (human-only), scored by API Evangelist on 2026-11-13.

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this

find_providersBrowse and filter every provider in the catalog.
get_provider_artifactsEvery artifact this provider publishes, grouped by type.
get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
get_provider_ratingPRO — composite, band, trend and facet scores.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This provider

curl "https://apis.io/api/v1/providers/pyspark"

All providers

curl "https://apis.io/api/v1/providers?limit=25"

Every operation they expose

curl "https://apis.io/api/v1/providers/pyspark/operations?limit=25"

How their score was established

curl "https://apis.io/api/v1/providers/pyspark/evidence"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.