Apache NiFi

Apache NiFi is a dataflow management system designed to automate the flow of data between systems. It provides a web-based user interface for designing, controlling, and monitoring data flows with real-time operational control, data provenance tracking, and support for hundreds of processors. NiFi Version 2 is the current major version with enhanced security and performance.

3 APIs 10 Features

Data IntegrationDataflowETLIoTStreamingData Pipeline

APIs

Apache NiFi REST API

The NiFi REST API provides comprehensive JWT-authenticated endpoints for managing processors, connections, controller services, process groups, reporting tasks, provenance, flow...

Apache NiFi Registry

NiFi Registry provides a central location for storage and management of shared NiFi flow resources, enabling versioned flows across NiFi environments. It provides its own REST A...

Apache MiNiFi

MiNiFi is a lightweight agent for edge data collection that is a subproject of NiFi. MiNiFi C++ (nifi-minifi-cpp) provides a small-footprint agent for IoT edge data collection w...

Collections

Apache NiFi REST API

OPEN

Pricing Plans

Apache Nifi Plans Pricing

3 plans

PLANS

Rate Limits

Apache Nifi Rate Limits

5 limits

RATE LIMITS

FinOps

Apache Nifi Finops

FINOPS

Features

Visual Dataflow Designer

Browser-based drag-and-drop interface for designing, controlling, and monitoring data flows without coding.

Data Provenance Tracking

Complete lineage tracking of every piece of data that flows through the system from ingestion to destination.

Hundreds of Built-in Processors

Extensive library of processors for data ingestion, transformation, routing, and delivery to diverse systems and cloud platforms.

Guaranteed Delivery

Loss-tolerant and guaranteed delivery options with configurable prioritization and backpressure control.

REST API

Comprehensive JWT-authenticated REST API for programmatic management of all NiFi resources and operations.

Flow Versioning

Version control for data flows via NiFi Registry, enabling flow promotion across development, test, and production environments.

Multi-Tenant Security

Fine-grained multi-tenant authorization with HTTPS, TLS, and SSH support for secure deployments.

Clustering

Zero-master cluster architecture for high-availability and load-balanced dataflow execution.

Parameter Contexts

Externalize configuration using parameter contexts that can be applied across multiple processors and process groups.

MiNiFi Edge Agents

Lightweight MiNiFi agents for edge data collection at IoT endpoints, managed centrally from NiFi.

Use Cases

Data Ingestion Pipelines

Build pipelines ingesting data from files, databases, message queues, cloud storage, and APIs into data lakes and warehouses.

Cybersecurity Data Collection

Collect and route security telemetry, logs, and threat intelligence feeds for SIEM and analytics platforms.

IoT Edge Data Collection

Deploy MiNiFi agents at IoT edge locations to collect, filter, and forward sensor data to central NiFi clusters.

Real-Time Event Streaming

Build event streaming pipelines consuming from Kafka, Kinesis, and other message brokers for real-time data processing.

Generative AI Data Pipelines

Build data preparation and vector database ingestion pipelines for generative AI and RAG applications.

Cloud Data Integration

Move and transform data between AWS, Azure, and GCP services with built-in cloud processor libraries.

Integrations

Apache Kafka

Native ConsumeKafka and PublishKafka processors for streaming data between NiFi and Kafka topics.

Amazon S3

PutS3Object and FetchS3Object processors for reading and writing data to AWS S3 buckets.

Azure Blob Storage

PutAzureBlobStorage and FetchAzureBlobStorage processors for Azure cloud storage integration.

Google Cloud Storage

Native GCS processors for reading and writing Google Cloud Storage objects.

MongoDB

PutMongo and GetMongo processors for reading and writing documents to MongoDB collections.

Elasticsearch

PutElasticsearchRecord and FetchElasticsearch processors for indexing and querying Elasticsearch.

Salesforce

Native Salesforce processors for querying and publishing data to Salesforce CRM.

Apache MQTT

ConsumeMQTT and PublishMQTT processors for IoT messaging protocol support.

Resources

GitHubOrganization

Sources

opencollection: 1.0.0
info:
  name: Apache NiFi REST API
  version: 2.x
items:
- info:
    name: Access
    type: folder
  items:
  - info:
      name: Create an access token using username and password
      type: http
    http:
      method: POST
      url: http://{host}:{port}/nifi-api/access/token
      body:
        type: form-urlencoded
        data:
        - name: username
          value: ''
        - name: password
          value: ''
    docs: Create an access token using username and password
  - info:
      name: Log the current user out and invalidate the JWT
      type: http
    http:
      method: DELETE
      url: http://{host}:{port}/nifi-api/access/logout
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Log the current user out and invalidate the JWT
  - info:
      name: Retrieve authentication configuration and status
      type: http
    http:
      method: GET
      url: http://{host}:{port}/nifi-api/authentication/configuration
    docs: Retrieve authentication configuration and status
- info:
    name: Connections
    type: folder
  items:
  - info:
      name: Retrieve a connection
      type: http
    http:
      method: GET
      url: http://{host}:{port}/nifi-api/connections/:id
      params:
      - name: id
        value: ''
        type: path
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Retrieve a connection
  - info:
      name: Update a connection
      type: http
    http:
      method: PUT
      url: http://{host}:{port}/nifi-api/connections/:id
      params:
      - name: id
        value: ''
        type: path
      body:
        type: json
        data: '{}'
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Update a connection
  - info:
      name: Delete a connection
      type: http
    http:
      method: DELETE
      url: http://{host}:{port}/nifi-api/connections/:id
      params:
      - name: id
        value: ''
        type: path
      - name: version
        value: ''
        type: query
        description: Revision version for optimistic locking
      - name: clientId
        value: ''
        type: query
      - name: disconnectedNodeAcknowledged
        value: ''
        type: query
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Delete a connection
- info:
    name: Controller Services
    type: folder
  items:
  - info:
      name: Retrieve a controller service
      type: http
    http:
      method: GET
      url: http://{host}:{port}/nifi-api/controller-services/:id
      params:
      - name: id
        value: ''
        type: path
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Retrieve a controller service
  - info:
      name: Update a controller service
      type: http
    http:
      method: PUT
      url: http://{host}:{port}/nifi-api/controller-services/:id
      params:
      - name: id
        value: ''
        type: path
      body:
        type: json
        data: '{}'
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Update a controller service
  - info:
      name: Delete a controller service
      type: http
    http:
      method: DELETE
      url: http://{host}:{port}/nifi-api/controller-services/:id
      params:
      - name: id
        value: ''
        type: path
      - name: version
        value: ''
        type: query
      - name: clientId
        value: ''
        type: query
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Delete a controller service
- info:
    name: Flow
    type: folder
  items:
  - info:
      name: Create a controller service within a process group
      type: http
    http:
      method: POST
      url: http://{host}:{port}/nifi-api/flow/process-groups/:id/controller-services
      params:
      - name: id
        value: ''
        type: path
      body:
        type: json
        data: '{}'
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Create a controller service within a process group
  - info:
      name: Retrieve overall flow status
      type: http
    http:
      method: GET
      url: http://{host}:{port}/nifi-api/flow/status
      auth:
        type: bearer
        token: '{{bearerToken}}'
    docs: Retrieve overall flow status
bundled: true