Promptfoo

aid: prompt-foo
name: Promptfoo
description: Promptfoo is an open-source CLI and TypeScript/Node.js library for evaluating, red-teaming, and security-testing
  LLM applications, agents, and RAG pipelines. It runs deterministic prompt evals with model-graded and rule-based assertions,
  generates dynamic adversarial attack probes across 50+ vulnerability categories (prompt injection, jailbreaks, RAG poisoning,
  PII leakage, harmful content, business rule violations), and integrates with CI/CD via a GitHub Action and a `code-scans`
  command for pull-request review. Promptfoo supports dozens of LLM providers — OpenAI, Anthropic Claude, Google Gemini, AWS
  Bedrock/SageMaker, Azure OpenAI, Mistral, Cohere, Groq, DeepSeek, Together, Fireworks, OpenRouter, LiteLLM, Vercel and Cloudflare
  AI gateways — plus local runtimes (Ollama, LocalAI, llama.cpp, vLLM, llamafile, Docker Model Runner) and custom HTTP/WebSocket/Python/JavaScript/Go/Ruby/Shell
  providers. It also ships an MCP server (`promptfoo mcp`), a Model Audit scanner (`scan-model`) for malicious ML artifacts,
  and a hosted Enterprise tier with team sharing, continuous monitoring, SSO, and a centralized compliance dashboard aligned
  to OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, and the EU AI Act. The project is MIT-licensed, has 21k+ GitHub stars, and
  is now part of OpenAI while remaining open source.
type: Index
accessModel:
  pricing: unknown
  onboarding: unknown
  trial: false
  try_now: false
  public: false
  label: Unknown
  confidence: low
  source: []
  generated: '2026-07-22'
  method: derived
position: Provider
access: 3rd-Party
image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/icons/prompt-foo.png
tags:
- LLM Evaluation
- LLM Red Teaming
- LLM Security
- AI Security
- Prompt Engineering
- Vulnerability Scanning
- Adversarial Testing
- Jailbreak Testing
- Prompt Injection
- RAG
- Agents
- MCP
- Model Audit
- OWASP LLM Top 10
- NIST AI RMF
- MITRE ATLAS
- EU AI Act
- CI/CD
- Open Source
- TypeScript
url: https://raw.githubusercontent.com/api-evangelist/prompt-foo/refs/heads/main/apis.yml
created: '2026-05-25'
modified: '2026-05-25'
specificationVersion: '0.20'
apis: []
common:
- type: TrustCenter
  url: security/prompt-foo-trust-center.yml
- type: DomainSecurity
  url: security/prompt-foo-domain-security.yml
- type: Website
  url: https://www.promptfoo.dev
- type: Portal
  url: https://www.promptfoo.dev
- type: Documentation
  url: https://www.promptfoo.dev/docs/intro/
- type: GettingStarted
  url: https://www.promptfoo.dev/docs/getting-started/
- type: Documentation
  name: CLI Reference
  url: https://www.promptfoo.dev/docs/usage/command-line/
- type: Documentation
  name: Red Team Quickstart
  url: https://www.promptfoo.dev/docs/red-team/quickstart/
- type: Documentation
  name: Providers
  url: https://www.promptfoo.dev/docs/providers/
- type: Documentation
  name: Assertions & Metrics
  url: https://www.promptfoo.dev/docs/configuration/expected-outputs/
- type: Documentation
  name: API Reference
  url: https://www.promptfoo.dev/docs/api-reference/
- type: Pricing
  url: https://www.promptfoo.dev/pricing/
- type: Blog
  url: https://www.promptfoo.dev/blog/
- type: StatusPage
  url: https://status.promptfoo.app
- type: TrustCenter
  url: https://trust.promptfoo.dev
- type: GitHubOrganization
  url: https://github.com/promptfoo
- type: SourceCode
  name: promptfoo (CLI + library)
  url: https://github.com/promptfoo/promptfoo
- type: SDKs
  name: Python wrapper for the promptfoo CLI
  url: https://github.com/promptfoo/promptfoo-python
- type: Tools
  name: Promptfoo GitHub Action
  url: https://github.com/promptfoo/promptfoo-action
- type: Tools
  name: ModelAudit — ML model security scanner
  url: https://github.com/promptfoo/modelaudit
- type: Tools
  name: MCP Agent Provider (custom provider for testing MCP servers)
  url: https://github.com/promptfoo/mcp-agent-provider
- type: Tools
  name: evil-mcp-server (red-team MCP target)
  url: https://github.com/promptfoo/evil-mcp-server
- type: CodeExamples
  name: example-app (industry/model/security demo app)
  url: https://github.com/promptfoo/example-app
- type: CodeExamples
  name: demo-app
  url: https://github.com/promptfoo/demo-app
- type: Library
  name: js-rouge (ROUGE summarization metric)
  url: https://github.com/promptfoo/js-rouge
- type: Package
  name: npm — promptfoo
  url: https://www.npmjs.com/package/promptfoo
- type: Package
  name: PyPI — promptfoo
  url: https://pypi.org/project/promptfoo/
- type: Package
  name: Homebrew — promptfoo
  url: https://formulae.brew.sh/formula/promptfoo
- type: HelmChart
  name: Helm chart for self-hosted Promptfoo
  url: https://github.com/promptfoo/promptfoo/tree/main/helm/chart/promptfoo
- type: License
  name: MIT License
  url: https://github.com/promptfoo/promptfoo/blob/main/LICENSE
- type: Forums
  name: Discord community
  url: https://discord.gg/promptfoo
- type: Twitter
  url: https://twitter.com/promptfoo
- type: LinkedIn
  url: https://www.linkedin.com/company/promptfoo
- type: ChangeLog
  url: https://github.com/promptfoo/promptfoo/releases
- type: Features
  data:
  - Open-source CLI and Node.js/TypeScript library for LLM evaluation and red teaming
  - '`promptfoo eval` — run deterministic prompt/model evals with caching, concurrency, and live reload'
  - '`promptfoo redteam` — generate and run dynamic adversarial probes across 50+ vulnerability categories'
  - '`promptfoo view` — local browser UI for browsing eval and red-team results'
  - '`promptfoo share` — publish a shareable URL for an eval or model audit'
  - '`promptfoo generate` — synthesize datasets, red-team tests, and assertions'
  - '`promptfoo optimize` — improve prompts against a target provider'
  - '`promptfoo scan-model` — security-scan ML model files (ModelAudit)'
  - '`promptfoo code-scans` — scan code changes for LLM security vulnerabilities in IDEs and CI/CD'
  - '`promptfoo mcp` — expose promptfoo tools as a Model Context Protocol server'
  - '`promptfoo retry`, `list`, `export`, `import`, `validate`, `debug`, `cache`, `auth`'
  - 'Assertions: rule-based (equals, contains, regex, javascript, python, cost, latency) and model-graded (llm-rubric, classifier,
    factuality, answer-relevance, similarity)'
  - Red-team plugins covering prompt injection, jailbreaks, PII leakage, harmful content, bias, business-rule violations,
    RAG poisoning, and agent/tool abuse
  - Attack strategies including multi-turn (Crescendo), GOAT (Meta), and iterative jailbreak techniques
  - 'Framework alignment: OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, EU AI Act'
  - '50+ providers: OpenAI, Anthropic, Google Gemini, AWS Bedrock, SageMaker, Azure OpenAI, Mistral, Cohere, Groq, DeepSeek,
    Together, Fireworks, Perplexity, OpenRouter, LiteLLM, Vercel AI Gateway, Cloudflare AI Gateway'
  - 'Local runtimes: Ollama, LocalAI, llama.cpp, vLLM, Docker Model Runner, llamafile'
  - Custom providers via HTTP, WebSocket, Python, JavaScript, Go, Ruby, and Shell
  - GitHub Action (`promptfoo/promptfoo-action`) for PR-level eval gating
  - Self-hostable via Helm chart and on-premise enterprise deployment
  - 'Hosted Enterprise tier: team sharing, continuous monitoring, centralized compliance dashboard, SSO, granular permissions,
    managed cloud, SLA-backed support'
  - SOC 2 and ISO 27001 certified; trust center at trust.promptfoo.dev
  - Distributed on npm (`promptfoo`), PyPI (`promptfoo`), and Homebrew (`brew install promptfoo`)
  - MIT-licensed; 21k+ GitHub stars; now part of OpenAI
  sources:
  - https://www.promptfoo.dev
  - https://github.com/promptfoo/promptfoo
  - https://www.promptfoo.dev/docs/usage/command-line/
  - https://www.promptfoo.dev/docs/red-team/quickstart/
  - https://www.promptfoo.dev/docs/providers/
  - https://www.promptfoo.dev/pricing/
  updated: '2026-05-25'
maintainers:
- FN: Kin Lane
  email: kin@apievangelist.com