Optimizely Holdouts API
The Holdouts API from Optimizely — 4 operation(s) for holdouts.
Work with this as data
Every API here is available over the APIs.io API and to AI agents over MCP.
MCP server
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
Tools for apis
7 MCP tools reach this
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.
Call it yourself
curl for this page
This API
curl "https://apis.io/api/v1/apis/optimizely-holdouts-api"
All apis
curl "https://apis.io/api/v1/apis?limit=25"Discovery needs no key. Ratings and market analysis are Pro.
OpenAPI Specification
openapi: 3.2.0
info:
title: Optimizely Holdouts API
x-refined-note:
- x-logo differs across the merged source definitions and was not carried
version: '1.0'
description: 'Operations tagged Holdouts across 2 of this provider''s published API definitions: optimizely-feature-experimentation-optimizely-feature-experimentation-api-openapi.json, optimizely-web-experimentation-optimizely-api-openapi.json. Each path carries the servers of the definition it was published in.'
servers:
- url: https://api.optimizely.com/flags/v1/
- url: https://api.optimizely.com/v2
security:
- apiKey: []
- OAuth2:
- read
- write
tags:
- name: Holdouts
paths:
/projects/{project_id}/holdouts:
get:
description: 'Get a list of Holdouts within a Project.
Holdouts are experiments that reserve a portion of your traffic to measure the impact of your feature flags and experiments. They help establish a control group by excluding a percentage of users from seeing any experiments, allowing you to measure the overall impact of your experimentation program.
You can use query parameters to filter and sort the response. Holdouts can be in different states (draft, running, concluded) and can be scoped globally or locally within your project.
'
operationId: list_holdouts
parameters:
- $ref: '#/components/parameters/projectId'
- description: 'Filter the Holdouts by archived status. Set to `true` to show only archived holdouts, `false` to show only active holdouts, or omit to show both.
'
in: query
name: archived
required: false
schema:
type: boolean
- description: 'Filter the Holdouts by status. Can be one or multiple values from: `draft`, `running`, or `concluded`. Multiple statuses can be provided as an array.
'
in: query
name: status
required: false
schema:
items:
$ref: '#/components/schemas/Holdout/properties/status'
type: array
- description: 'Search for Holdouts by name, description, or key. Performs a case-insensitive substring match on the holdout name, description, and key fields.
'
in: query
name: query
required: false
schema:
type: string
- description: 'Sort the Holdouts by one or more attributes. Use format `attribute:direction` where direction is either `asc` or `desc`. Multiple sort criteria are applied from left to right.
'
in: query
name: sort
schema:
items:
enum:
- name:asc
- name:desc
- key:asc
- key:desc
- created_time:asc
- created_time:desc
- updated_time:asc
- updated_time:desc
- status:asc
- status:desc
type: string
type: array
- $ref: '#/components/parameters/page'
- $ref: '#/components/parameters/perPage'
- $ref: '#/components/parameters/pageToken'
- $ref: '#/components/parameters/pageWindow'
responses:
'200':
$ref: '#/components/responses/HoldoutsPaginatedListSuccessResponse'
'400':
$ref: '#/components/responses/BadRequestResponse'
'401':
$ref: '#/components/responses/UnauthorizedResponse'
'403':
$ref: '#/components/responses/ForbiddenResponse'
'404':
$ref: '#/components/responses/NotFoundResponse'
summary: List Holdouts for a project
tags:
- Holdouts
post:
description: "Creates a new holdout experiment to establish a control group for measuring the impact of your feature flags and experiments.\n\nA holdout reserves a percentage of your traffic (specified in basis points) that will be excluded from seeing any experiments, allowing you to measure the overall impact of your experimentation program against a baseline control group.\n\nThe holdout will be created in a `draft` status and can be configured with:\n- **Traffic allocation**: Percentage of users to exclude from experiments (0-10000 basis points)\n- **Scope**: Global (affects all experiments) or local (specific experiments only) \n- **Audiences**: Target specific user segments using audience conditions\n- **Metrics**: Track and analyze the impact on key business metrics\n- **Environment**: Specify which environment the holdout applies to\n\n**Key Validation Rules:**\n- Holdout `key` must be unique within the project\n- Required fields: `key`, `name`\n- `traffic_allocation` must be between 0-10000 (basis points)\n- `key` must match pattern: `^[a-zA-Z0-9_\\-]+$` (max 64 characters)\n- `name` and `description` have character limits (255 and 1000 respectively)\n\n**Metrics Integration:** If metrics are provided, they will be automatically saved to the MetricsHub for tracking and analysis.\n"
operationId: create_holdout
parameters:
- $ref: '#/components/parameters/projectId'
requestBody:
$ref: '#/components/requestBodies/HoldoutRequest'
responses:
'201':
$ref: '#/components/responses/HoldoutSuccessResponse'
'400':
content:
application/json:
examples:
duplicate_key:
summary: Duplicate holdout key
value:
detail: Another entity already exists with the key 'checkout_holdout'.
status: 400
title: Key checkout_holdout already exists
type: about:blank
invalid_data:
summary: Invalid request data
value:
detail: traffic_allocation must be between 0 and 10000
status: 400
title: Invalid request data
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: "Bad Request - Invalid input data or validation failure.\n\nCommon causes:\n- Missing required fields (`key` or `name`)\n- Duplicate holdout key within the project \n- Invalid `key` format (must be alphanumeric with underscores/hyphens only)\n- `traffic_allocation` outside valid range (0-10000)\n- Field length limits exceeded\n- Invalid JSON structure in request body\n- Malformed audience conditions or metrics configuration\n"
'401':
$ref: '#/components/responses/UnauthorizedResponse'
'403':
$ref: '#/components/responses/ForbiddenResponse'
summary: Create a new Holdout
tags:
- Holdouts
servers:
- url: https://api.optimizely.com/flags/v1/
/projects/{project_id}/holdouts/{holdout_id}:
delete:
description: "Permanently and irreversibly delete a holdout by its unique ID. This is a destructive operation that cannot be undone.\n\n**⚠️ CAUTION - Permanent Deletion:**\n- All holdout configuration data will be permanently lost\n- Associated metric data will be deleted from MetricsHub \n- Experiment rules and historical data will be removed\n- There is **no way to recover** this data after deletion\n\n**What gets deleted:**\n- Holdout configuration (traffic allocation, audience conditions, etc.)\n- All associated experiment rules and their settings\n- Metric tracking data and historical performance \n- System metadata and lifecycle timestamps\n\n**Safer Alternative - Archiving:**\nIf you want to stop a holdout without permanent data loss, consider archiving instead:\n1. Update the holdout status to `concluded` (if currently running)\n2. Use the PATCH endpoint to set `archived: true`\n\nArchived holdouts preserve all historical data while removing them from active lists and preventing further modifications.\n\n**When to use DELETE vs ARCHIVE:**\n- **DELETE**: Use only when you're certain the holdout and all its data should be permanently removed\n- **ARCHIVE**: Recommended for most cases - preserves data for analysis while cleaning up active experiments\n\n**Important Notes:**\n- Can delete holdouts in any status (draft, running, concluded)\n- Operation affects both database and MetricsHub systems\n- This corresponds to the `delete_url` link in holdout responses\n"
operationId: delete_holdout
parameters:
- $ref: '#/components/parameters/projectId'
- $ref: '#/components/parameters/holdoutId'
responses:
'204':
content: {}
description: 'Successfully deleted the holdout. No response body is returned.
**Deletion completed:**
- Holdout configuration permanently removed from database
- Associated metrics deleted from MetricsHub
- All experiment rules and historical data destroyed
- Holdout is no longer accessible via any API endpoints
**What happens after deletion:**
- GET requests to this holdout will return 404 Not Found
- The holdout will not appear in list responses
- Associated URLs (update_url, fetch_url, etc.) become invalid
- Any references to this holdout in other systems should be cleaned up
A successful 204 response confirms the holdout has been completely removed from all systems.
'
'400':
content:
application/json:
example:
detail: Holdout ID must be a positive integer
status: 400
title: Invalid holdout ID
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: 'Bad Request - Invalid request parameters or malformed holdout ID.
**Common causes:**
- Invalid `holdout_id` format (must be a positive integer)
- Invalid `project_id` format (must be a positive integer)
- Malformed request structure
'
'401':
$ref: '#/components/responses/UnauthorizedResponse'
'403':
content:
application/json:
examples:
feature_disabled:
summary: Holdouts feature disabled
value:
detail: Holdouts feature is not enabled for this project
status: 403
title: Feature not enabled
type: about:blank
insufficient_permissions:
summary: Insufficient permissions
value:
detail: Permission denied - insufficient privileges to delete holdouts
status: 403
title: Forbidden
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: 'Forbidden - You don''t have permission to delete holdouts in this project.
**Permission requirements:**
- Must have project-level access
- Must have holdout deletion permissions
- Holdouts feature must be enabled for this project
**Common causes:**
- Insufficient user permissions for destructive operations
- Project-level permissions don''t include holdout management
- Organization-level policies restrict holdout deletion
- Holdouts feature flag is disabled for this project
'
'404':
content:
application/json:
examples:
holdout_not_found:
summary: Holdout does not exist
value:
detail: Holdout not found with id '987'
status: 404
title: Holdout not found
type: about:blank
project_not_found:
summary: Project does not exist
value:
detail: Project not found with id '12345'
status: 404
title: Project not found
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: "Not Found - The specified holdout does not exist or has already been deleted.\n\n**Possible reasons:**\n- Holdout ID does not exist in the specified project\n- Project ID does not exist or you don't have access to it\n- Holdout was already deleted by another user/process\n- Insufficient permissions to view or delete this holdout\n\n**Note**: This is also returned if the holdout exists but you lack delete permissions, \nfor security reasons (to prevent ID enumeration attacks).\n"
summary: Delete a Holdout
tags:
- Holdouts
get:
description: "Retrieves detailed information about a specific holdout by its unique ID within a project.\n\n**What you'll get:**\n- Complete holdout configuration including traffic allocation, status, and settings\n- Associated experiment rules and their current states \n- Audience conditions and targeting criteria (if configured)\n- Metric tracking configuration and current values\n- Lifecycle timestamps (created, updated, started, ended)\n- RESTful links for related operations (update, delete, archive, etc.)\n\n**Holdout States:**\n- **Draft**: Holdout is configured but not yet active\n- **Running**: Currently excluding traffic from experiments \n- **Concluded**: Finished collecting data, results available for analysis\n\n**Use Cases:**\n- View complete holdout configuration before making changes\n- Monitor current status and performance metrics\n- Access experiment rules affected by this holdout\n- Retrieve data for analysis and reporting\n- Verify permissions for management operations via included links\n\nThis operation corresponds to the `url` or `fetch_holdout_url` property found in holdout list responses.\n"
operationId: fetch_holdout
parameters:
- $ref: '#/components/parameters/projectId'
- $ref: '#/components/parameters/holdoutId'
responses:
'200':
content:
application/json:
examples:
concluded_holdout:
summary: Concluded holdout with complete lifecycle
value:
archive_url: /projects/12345/holdouts/archived
archived: false
created_time: '2024-06-28T10:00:00.000Z'
delete_url: /projects/12345/holdouts/987
description: Completed control group study showing 15% incremental impact
end_time: '2024-08-31T23:59:59.000Z'
environment_key: production
holdout_results_api_token: SampleToken.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
id: 987
key: q3_control_group
metrics:
- id: conversion_rate
name: Conversion Rate
type: custom
- id: revenue_per_visitor
name: Revenue per Visitor
type: custom
name: Q3 Marketing Control Group - Concluded
project_id: 12345
rules:
'456':
created_time: '2024-06-15T10:00:00.000Z'
flag_key: promo_banner
flag_name: Promotional Banner
flag_status: archived
rule_key: summer_promo
rule_name: Summer Promotion
traffic_allocation: 2000
updated_time: '2024-08-31T23:59:59.000Z'
start_time: '2024-07-01T09:00:00.000Z'
status: concluded
traffic_allocation: 1000
updated_time: '2024-08-31T23:59:59.000Z'
url: /projects/12345/holdouts/987
urn: holdouts.flag.optimizely.com::987
draft_holdout:
summary: Draft holdout ready for configuration
value:
archive_url: /projects/12345/holdouts/archived
archived: false
created_time: '2024-09-15T10:00:00.000Z'
delete_url: /projects/12345/holdouts/987
description: Control group for Q4 marketing campaigns to measure incremental impact
environment_key: production
holdout_results_api_token: SampleToken.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
id: 987
key: q4_control_group
metrics: []
name: Q4 Marketing Control Group
project_id: 12345
rules: {}
status: draft
traffic_allocation: 1000
update_url: /projects/12345/holdouts/987
updated_time: '2024-09-15T10:00:00.000Z'
url: /projects/12345/holdouts/987
urn: holdouts.flag.optimizely.com::987
running_holdout:
summary: Active holdout with experiment rules
value:
archived: false
created_time: '2024-09-15T10:00:00.000Z'
delete_url: /projects/12345/holdouts/987
description: Active control group excluding 10% traffic from all experiments
environment_key: production
holdout_results_api_token: SampleToken.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
id: 987
key: q4_control_group
metrics:
- id: revenue_impact
name: Revenue Impact
type: custom
name: Q4 Marketing Control Group
project_id: 12345
rules:
'456':
created_time: '2024-09-10T14:30:00.000Z'
flag_key: new_checkout
flag_name: New Checkout Experience
flag_status: unarchived
rule_key: checkout_experiment
rule_name: Checkout Flow Test
traffic_allocation: 2000
updated_time: '2024-09-12T11:15:00.000Z'
start_time: '2024-09-16T09:00:00.000Z'
status: running
traffic_allocation: 1000
update_url: /projects/12345/holdouts/987
updated_time: '2024-09-16T09:00:00.000Z'
url: /projects/12345/holdouts/987
urn: holdouts.flag.optimizely.com::987
schema:
$ref: '#/components/schemas/Holdout'
description: 'Successfully retrieved the holdout with all its configuration details.
**Response includes:**
- **Core properties**: ID, key, name, description, status, traffic allocation
- **Lifecycle data**: Created/updated timestamps, start/end times (if applicable)
- **Configuration**: Environment, audience conditions, metrics setup
- **Experiment rules**: Associated experiments and their allocation settings
- **Management links**: URLs for update, archive, delete operations (if authorized)
- **System metadata**: URN, project association, archive status
**Link availability depends on user permissions:**
- Missing `update_url`: Cannot modify this holdout
- Missing `delete_url`: Cannot permanently delete
- Missing `archive_url`/`unarchive_url`: Cannot change archive status
**Status-specific data:**
- **Draft holdouts**: Full configuration, no timing data
- **Running holdouts**: Includes `start_time`, active experiment rules
- **Concluded holdouts**: Includes both `start_time` and `end_time`
'
'400':
content:
application/json:
example:
detail: Holdout ID must be a positive integer
status: 400
title: Invalid holdout ID
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: "Bad Request - Invalid request parameters or malformed holdout ID.\n\n**Common causes:**\n- Invalid `holdout_id` format (must be a positive integer)\n- Invalid `project_id` format (must be a positive integer) \n- Malformed request structure or unsupported query parameters\n"
'401':
$ref: '#/components/responses/UnauthorizedResponse'
'403':
$ref: '#/components/responses/ForbiddenResponse'
'404':
content:
application/json:
example:
detail: Holdout not found with id '987'
status: 404
title: Holdout not found
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: 'Not Found - The specified holdout does not exist or you don''t have access to it.
**Possible reasons:**
- Holdout ID does not exist in the specified project
- Project ID does not exist or you don''t have access to it
- Holdout was permanently deleted
- Insufficient permissions to view this holdout
'
summary: Fetch a single Holdout
tags:
- Holdouts
post:
description: "Updates a specific holdout using JSON Patch operations (RFC 6902). This endpoint allows you to modify individual properties of an existing holdout without replacing the entire resource.\n\n**Supported Operations:**\n- **Status transitions**: Move holdouts through their lifecycle (`draft` → `running` → `concluded`)\n- **Configuration updates**: Modify traffic allocation, name, description, and other settings\n- **Archive management**: Archive/unarchive concluded holdouts\n- **Metrics updates**: Modify metric configurations (restricted based on status)\n\n**State Management & Validation Rules:**\n- **Draft holdouts**: All fields can be updated except readonly properties\n- **Running holdouts**: Limited updates allowed; cannot modify metrics or core configuration\n- **Concluded holdouts**: Only `name` and `archived` status can be modified\n- **Status transitions**: Must follow proper sequence (draft→running→concluded)\n- **Traffic allocation**: Must be 0-10000 basis points (0-100%)\n\n**Automatic Timestamp Management:**\n- Moving to `running` status automatically sets `start_time` to current timestamp\n- Moving to `concluded` status automatically sets `end_time` to current timestamp\n\n**Common Update Patterns:**\n```json\n[\n {\"op\": \"replace\", \"path\": \"/name\", \"value\": \"Updated Holdout Name\"},\n {\"op\": \"replace\", \"path\": \"/traffic_allocation\", \"value\": 1500},\n {\"op\": \"replace\", \"path\": \"/status\", \"value\": \"running\"}\n]\n```\n\n**Important Limitations:**\n- Cannot modify metrics on running holdouts (prevents mid-experiment changes)\n- Cannot archive non-concluded holdouts (must conclude first)\n- Cannot update core configuration on running/concluded holdouts\n- ReadOnly properties in patch operations are ignored\n"
operationId: update_holdouts
parameters:
- $ref: '#/components/parameters/projectId'
- $ref: '#/components/parameters/holdoutId'
requestBody:
content:
application/json-patch+json:
examples:
archive_holdout:
summary: Archive a concluded holdout
value:
- op: replace
path: /archived
value: true
conclude_holdout:
summary: Conclude a running holdout
value:
- op: replace
path: /status
value: concluded
multiple_updates:
summary: Update multiple properties at once
value:
- op: replace
path: /name
value: Q4 2024 Control Group
- op: replace
path: /description
value: Updated control group for Q4 campaign analysis
- op: replace
path: /traffic_allocation
value: 1000
start_holdout:
summary: Start a draft holdout (moves to running)
value:
- op: replace
path: /status
value: running
update_basic_properties:
summary: Update name and traffic allocation
value:
- op: replace
path: /name
value: Updated Control Group
- op: replace
path: /traffic_allocation
value: 2500
schema:
items:
$ref: '#/components/schemas/PatchRequestBody'
type: array
description: "Array of JSON Patch operations to apply to the holdout. Each operation specifies a change to make to the holdout resource.\n\n**Supported patch paths include:**\n- `/name` - Update holdout display name\n- `/description` - Update holdout description \n- `/status` - Change holdout status (draft/running/concluded)\n- `/traffic_allocation` - Update traffic percentage (0-10000 basis points)\n- `/archived` - Archive or unarchive the holdout\n- `/metrics` - Update metric configurations (not allowed on running holdouts)\n\n**Note**: Status transitions and archiving have specific validation rules enforced server-side.\n"
required: true
responses:
'200':
content:
application/json:
examples:
basic_update:
summary: Updated holdout with new name and traffic
value:
'987':
archive_url: /projects/12345/holdouts/archived
archived: false
created_time: '2019-05-08T00:00:00.000Z'
delete_url: /projects/12345/holdouts/987
description: Updated control group for better targeting
environment_key: production
holdout_results_api_token: SampleToken.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
id: 987
name: Updated Control Group
project_id: 12345
rules:
'456':
created_time: '2022-01-08T00:00:00.000Z'
flag_key: recurring_revenue
flag_name: Recurring Revenue
flag_status: unarchived
rule_key: experiment_one
rule_name: Experiment One
traffic_allocation: 2000
updated_time: '2022-01-08T00:00:00.000Z'
status: draft
traffic_allocation: 2500
unarchive_url: /projects/12345/holdouts/unarchived
update_url: /projects/12345/holdouts/987
updated_time: '2024-09-16T10:30:00.000Z'
url: /projects/12345/holdouts/987
urn: holdouts.flag.optimizely.com::987
started_holdout:
summary: Holdout moved to running status
value:
'987':
archived: false
created_time: '2019-05-08T00:00:00.000Z'
delete_url: /projects/12345/holdouts/987
description: Now actively excluding traffic from experiments
environment_key: production
holdout_results_api_token: SampleToken.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
id: 987
name: Live Control Group
project_id: 12345
start_time: '2024-09-16T10:30:00.000Z'
status: running
traffic_allocation: 1500
update_url: /projects/12345/holdouts/987
updated_time: '2024-09-16T10:30:00.000Z'
url: /projects/12345/holdouts/987
urn: holdouts.flag.optimizely.com::987
schema:
$ref: '#/components/schemas/HoldoutMap'
description: "Successful update of the holdout. Returns the updated holdout resource with all modifications applied.\n\n**Response Details:**\n- Contains the complete updated holdout with all applied changes\n- Includes RESTful JSON (https://restfuljson.org/) styled links for related actions\n- Automatic timestamps are included (start_time/end_time) if status was changed\n- Missing links indicate insufficient permissions for related operations\n- ReadOnly fields reflect any server-managed updates (like timestamps)\n\n**Status-Specific Behavior:**\n- **Draft→Running**: Includes newly set `start_time`\n- **Running→Concluded**: Includes newly set `end_time` \n- **Archived**: Shows `archived: true` with preserved historical data\n"
'400':
content:
application/json:
examples:
concluded_holdout_restriction:
summary: Limited updates on concluded holdout
value:
detail: Can't edit holdouts in 'concluded' state.
status: 400
title: Unable to update holdout
type: about:blank
invalid_state_transition:
summary: Invalid status transition
value:
detail: Holdout with key 'control_group' is not in 'draft' state, can't move to 'running' state.
status: 400
title: Unable to update holdout
type: about:blank
invalid_traffic_allocation:
summary: Traffic allocation out of range
value:
detail: Traffic allocation must be between 0 and 10000 (basis points).
status: 400
title: Unable to update holdout
type: about:blank
metrics_update_restricted:
summary: Cannot update metrics on running holdout
value:
detail: Can't update metrics on a running holdout.
status: 400
title: Unable to update holdout
type: about:blank
schema:
$ref: '#/components/schemas/ProblemDetail'
description: "Bad Request - Invalid patch operations, validation failures, or illegal state transitions.\n\n**Common causes:**\n- **Invalid state transitions**: Attempting invalid status changes (e.g., draft→concluded)\n- **Restricted updates**: Trying to modify metrics on running holdouts\n- **Invalid traffic allocation**: Values outside 0-10000 range\n- **Archive violations**: Attempting to archive non-concluded holdouts \n- **Malformed patch**: Invalid JSON Patch structure or unsupported operations\n- **Missing required fields**: Patch operations targeting required fields with null/empty values\n- **Type mismatches**: Providing wrong data types for specific fields\n"
'401':
$ref: '#/components/responses/UnauthorizedResponse'
'403':
$ref: '#/components/responses/ForbiddenResponse'
'404':
$ref: '#/components/responses/NotFoundResponse'
'409':
$ref: '#/components/responses/ConflictResponse'
summary: Update a Holdout using JSON Patch
tags:
- Holdouts
servers:
- url: https://api.optimizely.com/flags/v1/
/projects/{project_id}/holdouts/{holdout_id}/experiments:
get:
description: "Retrieves a list of experiments that are connected to a specific holdout through the system's data relationships.\n\n**What you'll get:**\n- **Experiment identification**: Layer experiment IDs from connected rules\n- **Timing information**: Start and end times from experiment
# --- truncated at 32 KB (82 KB total) ---
# Full source: https://raw.githubusercontent.com/api-evangelist/optimizely/refs/heads/main/openapi/optimizely-holdouts-api-openapi.yml