Apache Hudi

Apache Hudi is a data lake platform that provides incremental data processing primitives including upserts and incremental queries. It manages storage of large analytical datasets on distributed file systems with ACID transactions, timeline-based versioning, and integrations for Spark, Flink, and Hive.

2 APIs 8 Features

ACIDApacheBig DataData LakeIncremental ProcessingLakehouseOpen Source

APIs

Apache Hudi Timeline Server API

REST API for the Apache Hudi Timeline Server providing table timeline management, commit metadata inspection, and table administration for Hudi data lake tables.

Apache Hudi Java API

Java API for writing Hudi tables with upserts, inserts, and deletes, plus timeline management, compaction, and Spark/Flink DataSource integration APIs.

Collections

Apache Hudi Timeline Server API

OPEN

Pricing Plans

Apache Hudi Plans Pricing

3 plans

PLANS

Rate Limits

Apache Hudi Rate Limits

5 limits

RATE LIMITS

FinOps

Apache Hudi Finops

FINOPS

Features

ACID Upserts

Atomically insert or update records in data lake tables with ACID guarantees using record keys.

Hudi Timeline

Immutable commit timeline tracking all mutations for time travel, rollback, and incremental queries.

Incremental Queries

Query only the data changed since a given commit timestamp for efficient streaming ingestion.

Copy-On-Write Tables

COW table type rewrites entire Parquet files on upsert for read-optimized query performance.

Merge-On-Read Tables

MOR table type appends delta logs for fast writes with compaction-based read optimization.

Table Services

Built-in cleaning, compaction, clustering, and indexing services for table maintenance.

Multi-Engine Support

Read and write Hudi tables from Apache Spark, Flink, Hive, Presto, Trino, and Athena.

Schema Evolution

Support for adding, renaming, and dropping columns with backward-compatible schema evolution.

Use Cases

CDC Pipeline Ingestion

Ingest change data capture (CDC) events from databases into data lake tables with upsert support.

Streaming Data Lake

Build near-real-time data lake pipelines with Spark Structured Streaming or Flink.

Data Lake Maintenance

Manage storage costs with automated cleaning, compaction, and clustering of Hudi tables.

Incremental ETL

Build incremental ETL pipelines that process only changed data since the last run.

Regulatory Data Retention

Implement GDPR right-to-erasure by deleting records from Hudi tables with delete operations.

Semantic Vocabularies

Apache Hudi Timeline Context

25 classes · 0 properties

JSON-LD

API Governance Rules

Apache Hudi API Rules

8 rules · 2 errors 5 warnings 1 info

SPECTRAL

JSON Structure

Hudi Cleanconfig Structure

4 properties

JSON STRUCTURE

Hudi Commitmetadata Structure

6 properties

JSON STRUCTURE

Hudi Huditable Structure

7 properties

JSON STRUCTURE

Hudi Queryconfig Structure

4 properties

JSON STRUCTURE

Hudi Timelineinstant Structure

3 properties

JSON STRUCTURE

Hudi Writeconfig Structure

5 properties

JSON STRUCTURE

Example Payloads

Hudi Cleanconfig Example

4 fields

EXAMPLE

Hudi Commitmetadata Example

6 fields

EXAMPLE

Hudi Huditable Example

7 fields

EXAMPLE

Hudi Queryconfig Example

4 fields

EXAMPLE

Hudi Timelineinstant Example

3 fields

EXAMPLE

Hudi Writeconfig Example

5 fields

EXAMPLE

Resources

GitHubOrganization

Sources

opencollection: 1.0.0
info:
  name: Apache Hudi Timeline Server API
  version: 0.15.0
items:
- info:
    name: Timeline
    type: folder
  items:
  - info:
      name: Apache Hudi Get Timeline
      type: http
    http:
      method: GET
      url: http://localhost:9090/v1/hoodie/timeline/:tableName
      params:
      - name: tableName
        value: ''
        type: path
        description: Hudi table name
    docs: Get the complete timeline of instants for a Hudi table.
  - info:
      name: Apache Hudi Get Commit Metadata
      type: http
    http:
      method: GET
      url: http://localhost:9090/v1/hoodie/timeline/:tableName/commits/:commitTime
      params:
      - name: tableName
        value: ''
        type: path
      - name: commitTime
        value: ''
        type: path
        description: Commit timestamp
    docs: Get metadata for a specific commit instant on the Hudi timeline.
- info:
    name: Tables
    type: folder
  items:
  - info:
      name: Apache Hudi Create Table
      type: http
    http:
      method: POST
      url: http://localhost:9090/v1/hoodie/table
      body:
        type: json
        data: '{}'
    docs: Create a new Hudi table with the provided configuration.
  - info:
      name: Apache Hudi Get Table
      type: http
    http:
      method: GET
      url: http://localhost:9090/v1/hoodie/table/:tableName
      params:
      - name: tableName
        value: ''
        type: path
    docs: Get configuration and metadata for a Hudi table.
  - info:
      name: Apache Hudi Run Cleaning
      type: http
    http:
      method: POST
      url: http://localhost:9090/v1/hoodie/table/:tableName/clean
      params:
      - name: tableName
        value: ''
        type: path
      body:
        type: json
        data: '{}'
    docs: Trigger a cleaning operation on a Hudi table to remove old file versions.
bundled: true