Promptfoo

Promptfoo is an open-source LLM evaluation and red-teaming framework distributed as a TypeScript CLI and Node.js library under the MIT license. Developers use it to evaluate prompts, models, and RAG pipelines side by side, run automated red team attacks against LLM applications, scan code for LLM vulnerabilities in IDE and CI, and proxy Model Context Protocol traffic. Promptfoo also operates a commercial Enterprise platform at promptfoo.app for teams that need centralized governance, remediation reports, and shared evaluation history.

Promptfoo publishes 6 APIs on the APIs.io network. Tagged areas include LLM Evaluation, Red Teaming, AI Security, Guardrails, and Open Source.

Promptfoo’s developer surface includes documentation, API reference, getting-started guide, engineering blog, pricing, developer portal, tooling, and 24 more developer resources.

🌐 Visit website 📡 Source on GitHub

41.7/100 developing ▬ flat Agent 4/100 human only Front door AI 6/6 dominant open core · MIT Full breakdown ↓
scored 2026-10-04 · rubric v0.23.0

AccessFree

6 APIs 8 Features 5 Use Cases

LLM Evaluation Red Teaming AI Security Guardrails Open Source CLI Developer Tools

Kin Score

Kin Score How this is scored →
scored 2026-10-04 · rubric v0.23.0

Composite quality — 41.7/100 · developing

Contract Quality 1.2 / 19

Developer Ergonomics 7.9 / 15

Access Clarity 7.8 / 15

Operational Transparency 5.4 / 10

Contract Governance 0.0 / 9

Discoverability 4.7 / 8

Regulatory Posture 1.6 / 15

Open Source Surface 10.0 / 10

Agent readiness — 4/100 · human only

Machine-Readable Contract 0 / 18

Agentic Access Contract 0 / 10

Documented Reversibility 0 / 6

MCP Server 0 / 12

Machine-Readable Auth 0 / 10

Idempotency 0 / 9

Stable Error Semantics 0 / 8

Request/Response Examples 7 / 7

Rate-Limit Signaling 7 / 7

Typed Event Surface 0 / 6

Agent Skills 0 / 5

Well-Known Catalog 0 / 4

Consent & Bot Identity 0 / 3

A2A Agent Card 0 / 8

Dry-Run / Simulate Mode 0 / 4

Delegated User Identity 0 / 6

Protected Resource Metadata 0 / 5

Registration Without a Human 0 / 6

Agentic Commerce Well-Known Document 0 / 5

Regulatory Posture applies to this provider. Its tags matched the Horizontal (data, software, accessibility, platform) regime, so Regulatory Posture carries 15 points of the composite. If this regime is wrong for your business, say so on your provider repo — the applicability map is public and we will correct it.

Open Source Surface applies to this provider. This product is open source and we read its repository directly, so Open Source Surface carries 10 points of the composite. It is scored from what the repository actually publishes — a security policy, a contribution guide, a release history, a code of conduct — read live from the provider rather than inferred from our own catalog pointers. This facet adds; nothing was taken away to make room for it. An open-source project is not excused from the commercial facets, because exemption would strip it of the points it does earn. If we have the wrong repository, or this product is not open source, say so on your provider repo and we will drop the facet rather than have you publish against it.

Create-or-Update Ergonomics could not be measured. We hold no machine-readable contract for this provider to read, so there is nothing to measure a write surface against. Excluded rather than scored zero: never-measured and measured-empty are different facts. Publishing an OpenAPI is what makes this facet — and several others — scorable at all.

The six quality facets above are damped to 75 points between them, because both conditional facets apply and carry 25 points together. That is why each facet's contribution is shown against a damped maximum: raising a quality facet moves the composite by 75% of its nominal weight, not 100%. The full arithmetic is at apis.io/rating/.

Improve this rating by publishing the missing artifacts — every area above can be raised, and the full rubric is at apis.io/rating/. Every facet and dimension name above is a link: it opens that measurement's own page — what it means, the exact checks that feed it, how the whole catalog distributes on it, and the providers at the top of it. This rating is computed from github.com/api-evangelist/promptfoo: open an issue to ask a question, or submit a pull request to add artifacts. Submit an artifact on GitHub — free → Manage your own listing — the Influence plan, $499/mo →

APIs 6

Individual APIs this provider publishes, each with its own machine-readable definition.

Promptfoo CLI

The Promptfoo CLI is the primary entry point for running prompt and model evaluations from the command line or CI. It is installable via npm, Homebrew, pip, or npx and reads a Y...

Promptfoo Node.js Library

The Promptfoo Node.js package exposes the same evaluation engine programmatically so developers can embed evaluations, assertions, and dataset runs directly into JavaScript and ...

Promptfoo Red Team

Promptfoo Red Team generates adversarial test cases against LLM applications targeting prompt injection, jailbreaks, PII leakage, bias, and other OWASP LLM Top 10 categories. Ru...

Promptfoo Enterprise

Promptfoo Enterprise is the commercial SaaS at promptfoo.app providing centralized evaluation history, shared red team findings, remediation reports, role-based access control, ...

Promptfoo MCP Proxy

MCP Proxy is Promptfoo's security gateway for Model Context Protocol traffic, inspecting tool calls and responses flowing between agents and MCP servers to enforce policies and ...

Promptfoo Code Scanning

Promptfoo Code Scanning analyzes source code in IDEs and CI pipelines to find LLM-related vulnerabilities including unsafe prompt construction, missing guardrails, and risky too...

Pricing Plans 1

Published pricing tiers and plan structures.

Promptfoo Plans Pricing

1 plans

PLANS

Rate Limits 1

Documented rate limits and quota policies.

Promptfoo Rate Limits

2 limits

RATE LIMITS

FinOps 1

Cost, billing, and metering signals for API financial operations.

Promptfoo Finops

FINOPS

Features 8

Notable capabilities this provider offers.

Side-by-Side Evaluation

Compare prompts and models across providers with assertion-based scoring.

Red Teaming

Automated adversarial test generation across prompt injection, jailbreaks, PII, bias, and more.

RAG Evaluation

Evaluate retrieval-augmented generation pipelines for accuracy, faithfulness, and groundedness.

CI/CD Integration

Run evaluations on every pull request with exit-code-driven gating.

Local Web Viewer

Inspect evaluation runs and diffs in a local browser UI without sending data to a third party.

MCP Proxy

Inspect and govern Model Context Protocol traffic between agents and tool servers.

Code Scanning

Static analysis of LLM application code in IDEs and CI to surface risky patterns.

Self-Hosting

Open source CLI and library run locally with no telemetry to Promptfoo required.

Scroll for all 8

Security Posture 2

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Promptfoo Domain Security

TLSv1.3 · DMARC

SECURITY

Promptfoo Trust Center

SOC 2, ISO 27001

SECURITY

Use Cases 5

What developers build with this provider.

Prompt Iteration

Compare prompt variants against datasets to choose the best-performing version.

Model Selection

Benchmark candidate models across providers before committing to one in production.

RAG Quality Assurance

Evaluate chunking, retrieval, and generation choices in RAG systems.

AI Red Teaming

Probe pre-production LLM applications for adversarial failure modes.

Continuous LLM Testing

Wire evaluations and red team scans into CI so regressions block deploys.

Integrations 9

Pre-built integrations with other platforms and tools.

OpenAI

Native provider for OpenAI Chat Completions, Assistants, and Responses APIs.

Anthropic

Native provider for Anthropic Claude models.

Azure OpenAI

Provider for Azure-hosted OpenAI deployments.

AWS Bedrock

Provider for Anthropic, Meta, Mistral, and other models on Bedrock.

Google Vertex AI

Provider for Gemini and other models on Vertex.

Ollama

Provider for locally hosted open source models via Ollama.

LangChain

Evaluate LangChain chains and agents through custom providers.

GitHub Actions

Run promptfoo evaluations in CI on every pull request.

Model Context Protocol

MCP Proxy and MCP provider support for agentic workflows.

Scroll for all 9

Resources

Get Started 3

Portal, sign-up, and the first successful call

GettingStarted

GettingStarted

Login

Portal

Portal

Documentation 3

Reference material describing how the API behaves

ContributionGuide

ContributionGuide

Documentation

Documentation

APIReference

APIReference

Build 9

SDKs, sample code, and the tooling you integrate with

CodeOfConduct

CodeOfConduct

GitHubRepository

GitHubRepository

GitHubOrganization

SourceCode

SDKs

Tools

CodeExamples

Library

Package

Scroll for all 9

Access & Security 3

Authentication, authorization, and security posture

SecurityPolicy

SecurityPolicy

TrustCenter

TrustCenter

DomainSecurity

DomainSecurity

Operate 6

Status, limits, changes, and where to get help

IssueTracker

Discord

Contact

StatusPage

Forums

ChangeLog

Commercial 2

Pricing, plans, and the legal terms of use

Pricing

Pricing

License

License

Company 4

The organization behind the API

Website

Website

Blog

Blog

LinkedIn

Twitter

Twitter

Other 1

Properties that don't map to a standard resource type

HelmChart

HelmChart

Source (apis.yml)

aid: promptfoo
url: https://raw.githubusercontent.com/api-evangelist/promptfoo/refs/heads/main/apis.yml
name: Promptfoo
type: Index
deliveryModel:
  model: open-core
  license: MIT
  license_evidence:
    repo: promptfoo/promptfoo
    spdx: MIT
    stars: 25443
  open_source: true
  commercial: true
  callable_host: false
  label: Open core · an OSS project plus a commercial hosted product
  confidence: medium
  source:
  - pricing
  - repository-license
  generated: '2026-09-25'
  method: derived
accessModel:
  pricing: free
  onboarding: unknown
  trial: false
  try_now: false
  public: false
  label: Free
  confidence: medium
  source:
  - plans
  generated: '2026-07-22'
  method: derived
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/promptfoo.png
x-docs-platform: docusaurus
x-docs-platform-confidence: medium
x-docs-platform-checked: 2026-10-04
tags:
- LLM Evaluation
- Red Teaming
- AI Security
- Guardrails
- Open Source
- CLI
- Developer Tools
tags_raw:
- LLM Evaluation
- Red Teaming
- AI Security
- Guardrails
- Open Source
- CLI
- Developer Tools
- Open-Source
description: Promptfoo is an open-source LLM evaluation and red-teaming framework distributed as a TypeScript CLI and Node.js
  library under the MIT license. Developers use it to evaluate prompts, models, and RAG pipelines side by side, run automated
  red team attacks against LLM applications, scan code for LLM vulnerabilities in IDE and CI, and proxy Model Context Protocol
  traffic. Promptfoo also operates a commercial Enterprise platform at promptfoo.app for teams that need centralized governance,
  remediation reports, and shared evaluation history.
created: '2026-05-23'
modified: '2026-08-08'
specificationVersion: '0.23'
apis:
- aid: promptfoo:promptfoo-cli
  name: Promptfoo CLI
  tags:
  - CLI
  - LLM Evaluation
  - Open Source
  - Testing
  tags_raw:
  - CLI
  - LLM Evaluation
  - Open Source
  - Testing
  - Open-Source
  humanURL: https://www.promptfoo.dev/docs/getting-started/
  properties:
  - url: https://www.promptfoo.dev/docs/getting-started/
    type: GettingStarted
  - url: https://www.promptfoo.dev/docs/usage/command-line/
    type: Documentation
  - url: https://github.com/promptfoo/promptfoo
    type: SourceCode
  description: The Promptfoo CLI is the primary entry point for running prompt and model evaluations from the command line
    or CI. It is installable via npm, Homebrew, pip, or npx and reads a YAML configuration to fan out evaluations across providers
    including OpenAI, Anthropic, Azure, Bedrock, Google, and local Ollama models. Outputs include a local web viewer, JSON
    reports, and CI-friendly exit codes.
- aid: promptfoo:promptfoo-library
  name: Promptfoo Node.js Library
  tags:
  - SDK
  - Node.js
  - Library
  - LLM Evaluation
  humanURL: https://www.promptfoo.dev/docs/usage/node-package/
  properties:
  - url: https://www.promptfoo.dev/docs/usage/node-package/
    type: Documentation
  - url: https://www.npmjs.com/package/promptfoo
    type: SDKs
  description: The Promptfoo Node.js package exposes the same evaluation engine programmatically so developers can embed evaluations,
    assertions, and dataset runs directly into JavaScript and TypeScript applications and pipelines.
- aid: promptfoo:promptfoo-red-team
  name: Promptfoo Red Team
  tags:
  - Red Teaming
  - AI Security
  - Vulnerability Scanning
  - Adversarial Testing
  humanURL: https://www.promptfoo.dev/docs/red-team/
  properties:
  - url: https://www.promptfoo.dev/docs/red-team/
    type: Documentation
  description: Promptfoo Red Team generates adversarial test cases against LLM applications targeting prompt injection, jailbreaks,
    PII leakage, bias, and other OWASP LLM Top 10 categories. Runs from the CLI and produces structured findings that can
    be triaged in the Promptfoo viewer or Enterprise platform.
- aid: promptfoo:promptfoo-enterprise
  name: Promptfoo Enterprise
  tags:
  - Enterprise
  - Software-as-a-Service
  - Governance
  - Team Collaboration
  tags_raw:
  - Enterprise
  - SaaS
  - Governance
  - Team Collaboration
  humanURL: https://www.promptfoo.dev/docs/enterprise/
  properties:
  - url: https://www.promptfoo.dev/docs/enterprise/
    type: Documentation
  - url: https://promptfoo.app/
    type: ApplicationURL
  description: Promptfoo Enterprise is the commercial SaaS at promptfoo.app providing centralized evaluation history, shared
    red team findings, remediation reports, role-based access control, SSO, and team-wide governance over LLM evaluation workflows.
- aid: promptfoo:promptfoo-mcp-proxy
  name: Promptfoo MCP Proxy
  tags:
  - MCP
  - Proxy
  - AI Security
  tags_raw:
  - MCP
  - Proxy
  - AI Security
  - Model Context Protocol
  humanURL: https://www.promptfoo.dev/docs/usage/mcp/
  properties:
  - url: https://www.promptfoo.dev/docs/usage/mcp/
    type: Documentation
  description: MCP Proxy is Promptfoo's security gateway for Model Context Protocol traffic, inspecting tool calls and responses
    flowing between agents and MCP servers to enforce policies and surface adversarial activity.
- aid: promptfoo:promptfoo-code-scanning
  name: Promptfoo Code Scanning
  tags:
  - Code Scanning
  - IDE
  - CI/CD
  - Static Analysis
  humanURL: https://www.promptfoo.dev/docs/usage/code-scanning/
  properties:
  - url: https://www.promptfoo.dev/docs/usage/code-scanning/
    type: Documentation
  description: Promptfoo Code Scanning analyzes source code in IDEs and CI pipelines to find LLM-related vulnerabilities including
    unsafe prompt construction, missing guardrails, and risky tool usage in agent code.
common:
- type: IssueTracker
  url: https://github.com/promptfoo/promptfoo/issues
- type: SecurityPolicy
  url: https://github.com/promptfoo/promptfoo/blob/main/SECURITY.md
- type: CodeOfConduct
  url: https://github.com/promptfoo/promptfoo/blob/main/CODE_OF_CONDUCT.md
- type: ContributionGuide
  url: https://github.com/promptfoo/promptfoo/blob/main/CONTRIBUTING.md
- type: TrustCenter
  url: security/promptfoo-trust-center.yml
- type: DomainSecurity
  url: security/promptfoo-domain-security.yml
- type: Website
  url: https://www.promptfoo.dev/
- type: Documentation
  url: https://www.promptfoo.dev/docs/
- type: APIReference
  url: https://www.promptfoo.dev/docs/api-reference/
- type: GettingStarted
  url: https://www.promptfoo.dev/docs/getting-started/
- type: Blog
  url: https://www.promptfoo.dev/blog/
- type: Pricing
  url: https://www.promptfoo.dev/pricing/
- type: Login
  url: https://promptfoo.app/
- type: GitHubRepository
  url: https://github.com/promptfoo/promptfoo
- type: LinkedIn
  url: https://www.linkedin.com/company/promptfoo/
- type: Discord
  url: https://discord.gg/promptfoo
- type: Contact
  url: https://www.promptfoo.dev/contact/
- type: License
  url: https://github.com/promptfoo/promptfoo/blob/main/LICENSE
- type: Features
  data:
  - name: Side-by-Side Evaluation
    description: Compare prompts and models across providers with assertion-based scoring.
  - name: Red Teaming
    description: Automated adversarial test generation across prompt injection, jailbreaks, PII, bias, and more.
  - name: RAG Evaluation
    description: Evaluate retrieval-augmented generation pipelines for accuracy, faithfulness, and groundedness.
  - name: CI/CD Integration
    description: Run evaluations on every pull request with exit-code-driven gating.
  - name: Local Web Viewer
    description: Inspect evaluation runs and diffs in a local browser UI without sending data to a third party.
  - name: MCP Proxy
    description: Inspect and govern Model Context Protocol traffic between agents and tool servers.
  - name: Code Scanning
    description: Static analysis of LLM application code in IDEs and CI to surface risky patterns.
  - name: Self-Hosting
    description: Open source CLI and library run locally with no telemetry to Promptfoo required.
- type: UseCases
  data:
  - name: Prompt Iteration
    description: Compare prompt variants against datasets to choose the best-performing version.
  - name: Model Selection
    description: Benchmark candidate models across providers before committing to one in production.
  - name: RAG Quality Assurance
    description: Evaluate chunking, retrieval, and generation choices in RAG systems.
  - name: AI Red Teaming
    description: Probe pre-production LLM applications for adversarial failure modes.
  - name: Continuous LLM Testing
    description: Wire evaluations and red team scans into CI so regressions block deploys.
- type: Integrations
  data:
  - name: OpenAI
    description: Native provider for OpenAI Chat Completions, Assistants, and Responses APIs.
  - name: Anthropic
    description: Native provider for Anthropic Claude models.
  - name: Azure OpenAI
    description: Provider for Azure-hosted OpenAI deployments.
  - name: AWS Bedrock
    description: Provider for Anthropic, Meta, Mistral, and other models on Bedrock.
  - name: Google Vertex AI
    description: Provider for Gemini and other models on Vertex.
  - name: Ollama
    description: Provider for locally hosted open source models via Ollama.
  - name: LangChain
    description: Evaluate LangChain chains and agents through custom providers.
  - name: GitHub Actions
    description: Run promptfoo evaluations in CI on every pull request.
  - name: Model Context Protocol
    description: MCP Proxy and MCP provider support for agentic workflows.
- type: Portal
  url: https://www.promptfoo.dev
- type: StatusPage
  url: https://status.promptfoo.app
- type: GitHubOrganization
  url: https://github.com/promptfoo
- type: SourceCode
  name: promptfoo (CLI + library)
  url: https://github.com/promptfoo/promptfoo
- type: SDKs
  name: Python wrapper for the promptfoo CLI
  url: https://github.com/promptfoo/promptfoo-python
- type: Tools
  name: Promptfoo GitHub Action
  url: https://github.com/promptfoo/promptfoo-action
- type: CodeExamples
  name: example-app (industry/model/security demo app)
  url: https://github.com/promptfoo/example-app
- type: Library
  name: js-rouge (ROUGE summarization metric)
  url: https://github.com/promptfoo/js-rouge
- type: Package
  name: npm — promptfoo
  url: https://www.npmjs.com/package/promptfoo
- type: HelmChart
  name: Helm chart for self-hosted Promptfoo
  url: https://github.com/promptfoo/promptfoo/tree/main/helm/chart/promptfoo
- type: Forums
  name: Discord community
  url: https://discord.gg/promptfoo
- type: Twitter
  url: https://twitter.com/promptfoo
- type: ChangeLog
  url: https://github.com/promptfoo/promptfoo/releases
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com
x-merged-from:
- prompt-foo

Work with this as data

Every provider here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for providers

9 MCP tools reach this

find_providersBrowse and filter every provider in the catalog.
get_provider_artifactsEvery artifact this provider publishes, grouped by type.
get_provider_operationsEvery operation across all of their OpenAPIs — one call instead of parsing every spec.
get_provider_toolsEvery MCP tool they ship, with the operation each wraps.
get_provider_evidenceHow each part of their score was established. Free — the basis for a claim should not sit behind it.
get_provider_ratingPRO — composite, band, trend and facet scores.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This provider

curl "https://apis.io/api/v1/providers/promptfoo"

All providers

curl "https://apis.io/api/v1/providers?limit=25"

Every operation they expose

curl "https://apis.io/api/v1/providers/promptfoo/operations?limit=25"

How their score was established

curl "https://apis.io/api/v1/providers/promptfoo/evidence"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.