AI Gateway

An API Evangelist landscape index of AI gateways — the LLM routers, prompt firewalls, model fallback proxies, cost-control planes, and policy engines that sit between applications and AI providers. AI gateways unify access across OpenAI, Anthropic, Google, AWS Bedrock, Azure OpenAI, and self-hosted models behind a common interface and apply caching, routing, guardrails, observability, rate limiting, budgets, RBAC, and audit controls. This index catalogs commercial SaaS gateways, open-source projects, API gateway AI plugins, and cloud-provider AI proxies, with a shared schema and vocabulary for describing model routes, fallbacks, guardrails, and budgets across vendors.

15 APIs 13 Features

AI GatewayLLM RouterLLM ProxyModel RoutingPrompt FirewallGuardrailsAI ObservabilityCost ControlsAI GovernanceAPI Gateway

APIs

Portkey

Portkey is a production-grade AI gateway and control plane that fronts 1,600+ LLMs with unified routing, fallbacks, semantic caching, guardrails, cost attribution, and prompt ma...

OpenRouter

OpenRouter is a unified inference marketplace exposing 400+ models from 60+ providers behind one OpenAI-compatible API, with automatic provider fallback, pay-as-you-go credits, ...

LiteLLM

LiteLLM (BerriAI) is an open-source LLM gateway that exposes 100+ LLM providers — OpenAI, Anthropic, Azure, Bedrock, Gemini — through a single OpenAI-compatible API. The LiteLLM...

Helicone

Helicone is an open-source AI observability and routing platform centered on requests, sessions, prompts, datasets, rate limits, and alerts. Integrates with OpenAI, Anthropic, G...

Cloudflare AI Gateway

Cloudflare AI Gateway is an edge-deployed proxy that fronts AI providers — Workers AI, Anthropic, Google Gemini, OpenAI, Replicate, and more — with caching, rate limiting, analy...

Kong AI Gateway

The Kong AI Gateway is delivered as the AI Proxy plugin for Kong Gateway, transforming and proxying requests across 16+ providers including OpenAI, Azure OpenAI, Anthropic, Amaz...

Apache APISIX AI Proxy

The Apache APISIX ai-proxy plugin streamlines integration with LLMs by converting plugin settings into the appropriate request format for OpenAI, DeepSeek, Azure OpenAI, Anthrop...

Tetrate Agent Router Service

Tetrate Agent Router Service is an Envoy AI Gateway-as-a-service from the creators of Envoy, providing an approved LLM catalog, unified model access, automatic fallback, cost ma...

NVIDIA NIM

NVIDIA NIM is a set of inference microservices for streamlined AI model deployment, prebuilt and optimized for low-latency, high-throughput inference on NVIDIA-accelerated infra...

Traefik AI Gateway

Traefik AI Gateway is an enterprise, self-hosted, Kubernetes-native AI gateway with safety and governance (NVIDIA Safety NIMs, jailbreak detection, content filtering across 22+ ...

Together AI

Together AI is a full-stack AI Native Cloud for inference, fine-tuning, and GPU clusters powered by research, exposing serverless inference, batch processing, dedicated model an...

Anyscale

Anyscale is the production-scale AI platform built on Ray by the creators of Ray, supporting LLM inference and other data-intensive AI workloads across distributed GPU clusters....

LangDB

LangDB is an enterprise AI gateway for routing and governing LLM traffic across providers, with observability, cost tracking, and policy enforcement. Public homepage was unreach...

Envoy AI Gateway

Envoy AI Gateway is an open-source extension to Envoy Proxy and Envoy Gateway, providing a Kubernetes-native AI traffic plane for routing, governing, and observing LLM calls acr...

Gentrace

Gentrace was an AI evaluation and observability product; the company has shut down and its codebase is now MIT-licensed open source on GitHub. Included here for historical compl...

Collections

Portkey AI Gateway API

OPEN

Features

Provider Abstraction

A unified, typically OpenAI-compatible API surface that lets clients call any supported LLM provider without provider-specific SDK juggling.

Model Routing

Route requests to the right model and provider based on alias, header, request content, identity, time-of-day, cost, or latency.

Fallback and Failover

Automatically retry failed requests against backup providers or models when a primary upstream is degraded, rate-limited, or down.

Load Balancing and Fanout

Distribute traffic across multiple providers or replicas using weighted, priority-based, or RPM/TPM-aware load balancing.

Response Caching

Exact-match and semantic caching of model responses to cut latency and provider spend; some gateways claim 40-70 percent cost savings.

Cost Controls and Budgets

Per-user, per-team, per-key, per-project budgets, spend tracking, and hard or soft caps on token consumption.

Rate Limiting and Quotas

RPM, TPM, concurrency, and per-key quotas enforced at the gateway, decoupled from each upstream provider's limits.

Guardrails and Prompt Firewall

Prompt injection detection, jailbreak filtering, content moderation, PII redaction, and topic control applied to requests and responses.

Observability

Request, response, token, cost, latency, error, and trace data exported via OpenTelemetry, Langfuse, Phoenix, Langsmith, or built-in dashboards.

Authentication and RBAC

Virtual keys, JWT, OAuth2, SSO, and role-based access control over which clients can use which models with which budgets.

BYOK and Secret Management

Bring-your-own provider API keys, with the gateway holding and injecting them so clients never see upstream credentials.

Multi-Tenant Governance

Per-tenant isolation of keys, budgets, logs, and policies for platform teams serving multiple internal product teams.

MCP Federation

Some AI gateways also front Model Context Protocol servers, aggregating tools and exposing a single MCP endpoint to agents.

Use Cases

Provider-Agnostic LLM Access

Front many LLM providers behind one API so application teams can switch models without changing client code.

Cost Containment for AI

Apply caching, routing to cheaper models, and per-team budgets to keep generative-AI spend predictable.

Reliability and Failover

Survive single-provider outages by automatically failing over to backup models when the primary degrades.

Centralized AI Governance

Enforce content, PII, and policy controls in one place for every AI request leaving the organization.

Observability and FinOps

Attribute cost and latency to teams, projects, and users; expose token-level metrics to FinOps and SRE.

Multi-Tenant AI Platforms

Build internal AI platforms where each product team gets its own virtual keys, budgets, and logs.

Integrations

OpenAI

Front OpenAI's GPT, embeddings, and image models behind the gateway with virtual keys and budgets.

Anthropic

Route Claude requests through the gateway for fallback, caching, and central observability.

Google Gemini and Vertex AI

Proxy Google Gemini and Vertex AI calls with OpenAI-format translation where supported.

AWS Bedrock

Bridge OpenAI-format clients to Bedrock-hosted Anthropic, Mistral, Cohere, Meta, and Amazon models.

Azure OpenAI

Route to Azure-hosted OpenAI deployments with per-region failover and key rotation.

Ollama and vLLM

Front self-hosted Ollama and vLLM inference servers for hybrid cloud and on-prem inference.

OpenTelemetry

Export request, token, cost, and trace data to any OTel-compatible observability backend.

Langfuse and Phoenix

Stream prompts, completions, and evaluations to Langfuse and Arize Phoenix for prompt and model analytics.

Model Context Protocol

Some AI gateways federate MCP servers alongside LLM routes, exposing a unified agent endpoint.

JSONSchema

🔗

AI Gateway Provider Schema

JSONSchema

🔗

AI Gateway Policy Schema

JSONSchema

🔗

AI Gateway Route Structure

JSONStructure

🔗

AI Gateway Provider Structure

JSONStructure

🔗

AI Gateway Policy Structure

Sources

opencollection: 1.0.0
info:
  name: Portkey AI Gateway API
  version: 1.0.0
request:
  auth:
    type: apikey
    key: x-portkey-api-key
    value: '{{x-portkey-api-key}}'
    placement: header
items:
- info:
    name: Chat
    type: folder
  items:
  - info:
      name: Create a chat completion
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/chat/completions
    docs: Create a chat completion
- info:
    name: Completions
    type: folder
  items:
  - info:
      name: Create a text completion
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/completions
    docs: Create a text completion
- info:
    name: Embeddings
    type: folder
  items:
  - info:
      name: Create embeddings
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/embeddings
    docs: Create embeddings
  - info:
      name: Rerank documents
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/rerank
    docs: Rerank documents
- info:
    name: Images
    type: folder
  items:
  - info:
      name: Generate images
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/images/generations
    docs: Generate images
- info:
    name: Audio
    type: folder
  items:
  - info:
      name: Text-to-speech
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/audio/speech
    docs: Text-to-speech
  - info:
      name: Speech-to-text transcription
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/audio/transcriptions
    docs: Speech-to-text transcription
  - info:
      name: Audio translation
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/audio/translations
    docs: Audio translation
- info:
    name: Batches
    type: folder
  items:
  - info:
      name: List batches
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/batches
    docs: List batches
  - info:
      name: Create a batch job
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/batches
    docs: Create a batch job
  - info:
      name: Retrieve a batch
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/batches/:id
      params:
      - name: id
        value: ''
        type: path
    docs: Retrieve a batch
  - info:
      name: Cancel a batch
      type: http
    http:
      method: DELETE
      url: https://api.portkey.ai/v1/batches/:id
      params:
      - name: id
        value: ''
        type: path
    docs: Cancel a batch
- info:
    name: Files
    type: folder
  items:
  - info:
      name: List files
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/files
    docs: List files
  - info:
      name: Upload a file
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/files
    docs: Upload a file
  - info:
      name: Get file content
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/files/:id/content
      params:
      - name: id
        value: ''
        type: path
    docs: Get file content
- info:
    name: FineTuning
    type: folder
  items:
  - info:
      name: List fine-tuning jobs
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/fine_tuning/jobs
    docs: List fine-tuning jobs
  - info:
      name: Create a fine-tuning job
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/fine_tuning/jobs
    docs: Create a fine-tuning job
  - info:
      name: List events for a fine-tuning job
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/fine_tuning/jobs/:id/events
      params:
      - name: id
        value: ''
        type: path
    docs: List events for a fine-tuning job
- info:
    name: Assistants
    type: folder
  items:
  - info:
      name: List assistants
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/assistants
    docs: List assistants
  - info:
      name: Create an assistant
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/assistants
    docs: Create an assistant
- info:
    name: Threads
    type: folder
  items:
  - info:
      name: Create a thread
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/threads
    docs: Create a thread
  - info:
      name: List messages in a thread
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/threads/:id/messages
      params:
      - name: id
        value: ''
        type: path
    docs: List messages in a thread
  - info:
      name: Add a message to a thread
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/threads/:id/messages
      params:
      - name: id
        value: ''
        type: path
    docs: Add a message to a thread
  - info:
      name: Run an assistant against a thread
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/threads/:id/runs
      params:
      - name: id
        value: ''
        type: path
    docs: Run an assistant against a thread
- info:
    name: Prompts
    type: folder
  items:
  - info:
      name: Execute a saved prompt template
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/prompts/completions
    docs: Execute a saved prompt template
  - info:
      name: Render a prompt template with variables
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/prompts/render
    docs: Render a prompt template with variables
  - info:
      name: List prompts
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/prompts
    docs: List prompts
  - info:
      name: Create a prompt
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/prompts
    docs: Create a prompt
  - info:
      name: List prompt collections
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/prompts/collections
    docs: List prompt collections
  - info:
      name: List prompt labels
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/prompts/labels
    docs: List prompt labels
  - info:
      name: List reusable prompt partials
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/prompts/partials
    docs: List reusable prompt partials
- info:
    name: Responses
    type: folder
  items:
  - info:
      name: Create a response (Open Responses spec)
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/responses
    docs: Create a response (Open Responses spec)
- info:
    name: Analytics
    type: folder
  items:
  - info:
      name: Time-series analytics data
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/analytics/graphs/
    docs: Time-series analytics data
  - info:
      name: Grouped analytics data
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/analytics/groups/
    docs: Grouped analytics data
- info:
    name: Configs
    type: folder
  items:
  - info:
      name: List gateway configurations
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/configs
    docs: List gateway configurations
  - info:
      name: Create a gateway configuration
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/configs
    docs: Create a gateway configuration
- info:
    name: APIKeys
    type: folder
  items:
  - info:
      name: List API keys
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/api-keys
    docs: List API keys
  - info:
      name: Create an API key
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/api-keys
    docs: Create an API key
- info:
    name: Integrations
    type: folder
  items:
  - info:
      name: List LLM provider integrations
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/integrations
    docs: List LLM provider integrations
  - info:
      name: Create an integration
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/integrations
    docs: Create an integration
  - info:
      name: List providers
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/providers
    docs: List providers
- info:
    name: VirtualKeys
    type: folder
  items:
  - info:
      name: List virtual keys
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/virtual-keys
    docs: List virtual keys
  - info:
      name: Create a virtual key
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/virtual-keys
    docs: Create a virtual key
- info:
    name: Guardrails
    type: folder
  items:
  - info:
      name: List guardrails
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/guardrails
    docs: List guardrails
  - info:
      name: Create a guardrail policy
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/guardrails
    docs: Create a guardrail policy
- info:
    name: Policies
    type: folder
  items:
  - info:
      name: List rate-limit policies
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/policies/rate-limits
    docs: List rate-limit policies
  - info:
      name: Create a rate-limit policy
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/policies/rate-limits
    docs: Create a rate-limit policy
  - info:
      name: List usage-limit policies
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/policies/usage-limits
    docs: List usage-limit policies
  - info:
      name: Create a usage-limit policy
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/policies/usage-limits
    docs: Create a usage-limit policy
- info:
    name: Workspaces
    type: folder
  items:
  - info:
      name: List workspaces
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/workspaces
    docs: List workspaces
  - info:
      name: Create a workspace
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/workspaces
    docs: Create a workspace
  - info:
      name: List workspace members
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/workspace-members
    docs: List workspace members
  - info:
      name: Invite a user to the workspace
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/user-invites
    docs: Invite a user to the workspace
  - info:
      name: List users
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/users
    docs: List users
- info:
    name: MCP
    type: folder
  items:
  - info:
      name: List MCP integrations
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/mcp-integrations
    docs: List MCP integrations
  - info:
      name: List MCP server instances
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/mcp-servers
    docs: List MCP server instances
- info:
    name: Logs
    type: folder
  items:
  - info:
      name: Query request logs
      type: http
    http:
      method: GET
      url: https://api.portkey.ai/v1/logs
    docs: Query request logs
  - info:
      name: Insert request logs
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/logs
    docs: Insert request logs
  - info:
      name: Create a log export
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/logs/exports
    docs: Create a log export
- info:
    name: Feedback
    type: folder
  items:
  - info:
      name: Submit user feedback for a request
      type: http
    http:
      method: POST
      url: https://api.portkey.ai/v1/feedback
    docs: Submit user feedback for a request
bundled: true

AI Gateway

APIs

Portkey

OpenRouter

LiteLLM

Helicone

Cloudflare AI Gateway

Kong AI Gateway

Apache APISIX AI Proxy

Tetrate Agent Router Service

NVIDIA NIM

Traefik AI Gateway

Together AI

Anyscale

LangDB

Envoy AI Gateway

Gentrace

Collections

Portkey AI Gateway API

Features

Use Cases

Integrations

Semantic Vocabularies

Ai Gateway Context

JSON Structure

Ai Gateway Policy Structure

Ai Gateway Provider Structure

Ai Gateway Route Structure

Example Payloads

Ai Gateway Cost Route Example

Ai Gateway Fanout Route Example

Ai Gateway Policy Budget Example

Ai Gateway Policy Pii Example

Ai Gateway Provider Example

Ai Gateway Route Example

Resources

Sources