statsig Experiments API
Manage A/B test experiments including creation, configuration, starting, resetting, and analysis.
Operations 66
Each operation below carries the questions people ask an LLM about it and the instructions they give an agent to run it. Generated by API Evangelist overlay
Ask an LLM
“Which experiments are running or set up in my Statsig project?”
“Can I see the status of every experiment we have, a page at a time?”
Tell an agent
List all experiments in my project with their status.
Show page {page} of experiments, {limit} per page.
Ask an LLM
“How do I set up a new A/B test with a hypothesis and groups?”
“Can I put a new experiment inside a layer or behind a targeting gate?”
Tell an agent
Create an experiment named {name} with hypothesis {hypothesis}.
Create experiment {name} in layer {layerID} allocating {allocation} percent of users.
Ask an LLM
“What groups, parameters and hypothesis does a specific experiment have?”
“Can I look up how an experiment is configured in the console by its name?”
Tell an agent
Show the console configuration of experiment {id}.
Get the groups and hypothesis defined for experiment {id}.
Ask an LLM
“Can I change the allocation of an experiment that already exists?”
“How do I edit the hypothesis or tags on an existing experiment?”
Tell an agent
Change the allocation of experiment {id} to {allocation} percent.
Update the hypothesis of existing experiment {id} to {hypothesis}.
Ask an LLM
“How do I permanently remove an experiment from the project?”
“Is there a way to delete an old experiment we no longer need?”
Tell an agent destructive · confirm first
Delete experiment {id}.
Permanently remove experiment {id} from the project.
Ask an LLM
“How do I start an experiment so users begin getting assigned to groups?”
“When does an experiment begin logging exposures?”
Tell an agent
Start experiment {id}.
Kick off experiment {id} and begin allocating users to its groups.
Ask an LLM
“Can I wipe an experiment's collected data and start its assignments over?”
“What happens to user assignments when an experiment is reset?”
Tell an agent destructive · confirm first
Reset experiment {id}, clearing its collected data.
Clear all results and user assignments for experiment {id}.
Ask an LLM
“Which experiment group is a particular user assigned to?”
“Does fetching a user's experiment assignment log an exposure?”
Tell an agent
Get the group assignment for user {user} in experiment {experimentName}.
Evaluate experiment {experimentName} for user {user} and return their parameters.
Ask an LLM
“Which experiments are stale or still in setup, filtered by team or tag?”
“Can I find every experiment a particular creator launched within a date range?”
Tell an agent
Find experiments with status {status} tagged {tags}.
List console v1 experiments in layer {layerID} created between {createdStartDate} and {createdEndDate}.
Ask an LLM
“Can I turn on sequential testing or Bonferroni correction when I create an experiment?”
“Is it possible to set primary and secondary metrics and a control group for a new test in the v1 console?”
Tell an agent
Create a console v1 experiment {name} with primary metrics {primaryMetrics} and control group {controlGroupID}.
Set up new experiment {name} running {duration} days with sequential testing {sequentialTesting}.
Ask an LLM
“Can I pull back only certain fields of an experiment, like its metrics and status?”
“Where do I read the full console v1 record for one experiment, analysis settings included?”
Tell an agent
Fetch console v1 experiment {id} returning only the fields {fields}.
Read the full v1 experiment record for {id}, including its analysis settings.
POST
/console/v1/experiments/{id}
Replace an experiment's entire configuration · Fully Update Experiment
#
Ask an LLM
“How can I overwrite every setting on an experiment in one request?”
“Which fields must I resend when replacing an experiment's whole definition?”
Tell an agent
Replace the whole configuration of experiment {id}, resending groups {groups} and allocation {allocation}.
Fully overwrite experiment {id} with hypothesis {hypothesis} and status {status}.
PATCH
/console/v1/experiments/{id}
Change selected settings on an experiment · Partially Update Experiment
#
Ask an LLM
“Can I switch on turbo mode or change the confidence interval of a running experiment?”
“Is it possible to edit just the secondary metrics of an experiment without resending the rest?”
Tell an agent
Patch experiment {id} so its secondary metrics are {secondaryMetrics}.
Set the default confidence interval of experiment {id} to {defaultConfidenceInterval}.
DELETE
/console/v1/experiments/{id}
Delete an experiment via the Console v1 API · Delete Experiment
#
Ask an LLM
“Can the console v1 endpoint erase an experiment for good?”
“What is the v1 call to drop an experiment from the console entirely?”
Tell an agent destructive · confirm first
Erase experiment {id} using the console v1 endpoint.
Drop v1 experiment {id} from the console for good.
Ask an LLM
“How do I abandon an experiment without shipping any group?”
“Do I have to give a reason when abandoning a test?”
Tell an agent destructive · confirm first
Abandon experiment {id} because {decisionReason}.
Give up on experiment {id} with the reason {decisionReason}.
GET
/console/v1/experiments/{id}/alerts
Check an experiment's guardrail alerts · Get Experiment Guardrail Alert Statuses
#
Ask an LLM
“Have any guardrail metrics fired alerts on my experiment?”
“Which guardrail alerts are currently open for a test?”
Tell an agent
Show the guardrail alert statuses for experiment {id}.
Check whether any guardrail alerts have fired on experiment {id}.
POST
/console/v1/experiments/{id}/alerts/{metricId}/resolve
Resolve a metric rollout alert · Resolve Metric Rollout Alert
#
Ask an LLM
“How do I mark a metric rollout alert on an experiment as resolved?”
“Can I record my reasoning when dismissing a guardrail alert for one metric?”
Tell an agent
Resolve the rollout alert for metric {metricId} on experiment {id}.
Mark metric {metricId}'s alert on experiment {id} resolved, noting {reasoning}.
Ask an LLM
“How do I archive a finished experiment so it leaves the active list?”
“Can I note why an experiment was archived?”
Tell an agent
Archive experiment {id}.
Archive experiment {id} with the reason {archiveReason}.
POST
/console/v1/experiments/{id}/code_cleanup
Start code cleanup for an experiment · Start Experiment Code Cleanup
#
Ask an LLM
“Can Statsig kick off cleanup of the code paths behind a concluded experiment?”
“How do I start removing dead experiment code once a test is done?”
Tell an agent
Start code cleanup for experiment {id}.
Begin removing the leftover code references for experiment {id}.
Ask an LLM
“What background context is recorded for an experiment?”
“Can I fetch the context summary behind a test before reviewing results?”
Tell an agent
Get the context for experiment {id}.
Pull the background context recorded on experiment {id}.
GET
/console/v1/experiments/{id}/cumulative_exposures
Get cumulative exposures for an experiment · Retrieve cumulative exposures
#
Ask an LLM
“How many users have been exposed to my experiment so far in total?”
“Can I see exposure counts accumulating over the life of a test?”
Tell an agent
Show cumulative exposures for experiment {id}.
Get the running total of users exposed to experiment {id}.
PUT
/console/v1/experiments/{id}/defer_decision
Conclude an experiment and defer the decision · Conclude Experiment & Defer Decision
#
Ask an LLM
“Can I stop an experiment now but decide which group to ship later?”
“Is there a way to end a test without picking a winner yet?”
Tell an agent
Conclude experiment {id} and defer the ship decision.
Stop experiment {id} for the reason {stopReason} and decide on a winner later.
GET
/console/v1/experiments/{id}/diagnostics_checks
Get an experiment's health check diagnostics · Retrieve Experiment Checks Diagnostics
#
Ask an LLM
“Are there sample ratio mismatch or other health check failures on my experiment?”
“Can I see experiment diagnostics checks for just the last few days?”
Tell an agent
Show health check diagnostics for experiment {id} over the last {lastDays} days.
Get {type} diagnostics checks for experiment {id} from {startDate} to {endDate}.
GET
/console/v1/experiments/{id}/dimensional_exposures
Break down experiment exposures by dimension · Retrieve Exposures By Dimension
#
Ask an LLM
“Are exposures to my experiment balanced across dimensions like platform or country?”
“Which dimensions show severe exposure imbalances in a test?”
Tell an agent
Break down exposures for experiment {id} by dimension {dimension_type}.
Show dimensional exposure imbalances of severity {severity} for experiment {id}.
POST
/console/v1/experiments/{id}/disable_groups
Disable groups in an experiment · Disable Experiment Groups
#
Ask an LLM
“Can I switch off one variant of an experiment while the others keep running?”
“How do I stop sending users to a badly performing test group?”
Tell an agent
Disable groups {group_names} in experiment {id}.
Turn off the variant {group_names} on experiment {id} so no new users land in it.
GET
/console/v1/experiments/{id}/eligible_reviewers
List who can review an experiment · Get Eligible Reviewers
#
Ask an LLM
“Who is allowed to approve changes to my experiment?”
“Which teammates are eligible reviewers for a test?”
Tell an agent
List the eligible reviewers for experiment {id}.
Show who can approve review requests on experiment {id}.
POST
/console/v1/experiments/{id}/enable_groups
Re-enable groups in an experiment · Enable Experiment Groups
#
Ask an LLM
“How do I turn a disabled experiment group back on?”
“Can a variant I switched off start receiving users again?”
Tell an agent
Enable groups {group_names} in experiment {id}.
Re-enable the previously disabled variant {group_names} on experiment {id}.
POST
/console/v1/experiments/{id}/load_pulse
Trigger a Pulse results load in the warehouse · Load Pulse (Warehouse Native)
#
Ask an LLM
“How do I kick off a fresh Pulse calculation for a warehouse native experiment?”
“Can I reload Pulse for only certain metrics, with turbo mode?”
Tell an agent
Load Pulse for warehouse native experiment {id}.
Run a Pulse load on experiment {id} for metrics {metricIDs} with turbo mode {turboMode}.
PUT
/console/v1/experiments/{id}/make_decision
Finish an experiment early and ship a group · Finish Experiment Early
#
Ask an LLM
“How do I end an experiment early and ship the winning group?”
“Can I keep a holdout or remove targeting when I make the ship decision?”
Tell an agent
Finish experiment {id} early and ship group {groupId} because {decisionReason}.
Ship group {groupId} of experiment {id} now, removing targeting {removeTargeting}, reason {decisionReason}.
GET
/console/v1/experiments/{id}/overrides
View an experiment's overrides · Get Experiment Overrides
#
Ask an LLM
“Which users or segments are forced into a particular experiment group?”
“What overrides are currently set on a test?”
Tell an agent
Show the overrides on experiment {id}.
List which users and segments are forced into groups in experiment {id}.
POST
/console/v1/experiments/{id}/overrides
Replace all overrides on an experiment · Update Experiment Overrides
#
Ask an LLM
“How do I replace the full set of overrides on an experiment at once?”
“Can I reset every forced user ID assignment in a test with a new list?”
Tell an agent
Replace all overrides on experiment {id} with {overrides} and user ID overrides {userIDOverrides}.
Overwrite the complete override list of experiment {id} with gate and segment overrides {overrides} and user IDs {userIDOverrides}.
PATCH
/console/v1/experiments/{id}/overrides
Add to an experiment's overrides · Partially Update Experiment Overrides
#
Ask an LLM
“Can I add a few user overrides to an experiment without replacing the existing ones?”
“How do I merge new segment overrides into a test's current list?”
Tell an agent
Add user ID overrides {userIDOverrides} and conditional overrides {overrides} to experiment {id}, keeping existing ones.
Merge segment overrides {overrides} and user overrides {userIDOverrides} into the current list on experiment {id}.
DELETE
/console/v1/experiments/{id}/overrides
Remove all overrides from an experiment · Delete Experiment Overrides
#
Ask an LLM
“How do I clear every override on an experiment at once?”
“Can I wipe all forced group assignments from a test?”
Tell an agent destructive · confirm first
Delete all overrides on experiment {id}.
Clear every forced group assignment from experiment {id}.
DELETE
/console/v1/experiments/{id}/overrides/conditional/{type}/{name}
Remove one gate or segment override · Delete a Single Experiment Override
#
Ask an LLM
“How do I remove a single segment override from an experiment group?”
“Can I drop one gate override in a specific environment and leave the rest?”
Tell an agent destructive · confirm first
Remove the {type} override {name} for group {groupID} on experiment {id}.
Delete the {type} override {name} from group {groupID} of experiment {id} in {environment}.
DELETE
/console/v1/experiments/{id}/overrides/userID/{userID}
Remove one user ID override · Delete a Single Experiment User ID Override
#
Ask an LLM
“How do I take one user out of a forced experiment group?”
“Can I un-override a single user ID for only one unit type and environment?”
Tell an agent destructive · confirm first
Remove user {userID} from the group {groupID} override on experiment {id}.
Stop forcing user {userID} into group {groupID} of experiment {id} for unit type {unitType}.
GET
/console/v1/experiments/{id}/pulse_load_history
List past Pulse loads for an experiment · Pulse Load History (Warehouse Native)
#
Ask an LLM
“When was Pulse last loaded for my warehouse native experiment?”
“Can I page through the history of Pulse loads on a test?”
Tell an agent
Show the Pulse load history for experiment {id}.
List page {page} of past Pulse loads for experiment {id}, {limit} at a time.
GET
/console/v1/experiments/{id}/pulse_load_history/{dagID}
Get details of one Pulse load run · Get Pulse Load History Details (Warehouse Native)
#
Ask an LLM
“Why did a particular Pulse load fail or take so long?”
“What steps ran in one specific warehouse Pulse load?”
Tell an agent
Get the details of Pulse load {dagID} for experiment {id}.
Show the status and steps of load run {dagID} on experiment {id}.
POST
/console/v1/experiments/{id}/pulse_load_history/{dagID}/cancel
Cancel a running Pulse load · Cancel Pulse Load (Warehouse Native)
#
Ask an LLM
“Can I stop a Pulse load that is still running in my warehouse?”
“How do I cancel a stuck warehouse Pulse computation?”
Tell an agent destructive · confirm first
Cancel Pulse load {dagID} for experiment {id}.
Abort the in-progress load run {dagID} on experiment {id}.
GET
/console/v1/experiments/{id}/pulse_metric_result
Get Pulse results for a single metric · Retrieve Pulse Metric Result
#
Ask an LLM
“What lift does one specific metric show between test and control?”
“Can I see a single metric's result with CUPED applied?”
Tell an agent
Get the Pulse result for metric {metricID} in experiment {id}, comparing {test} to {control}.
Show metric {metricID}'s lift for {test} vs {control} on experiment {id} at {confidence} confidence.
GET
/console/v1/experiments/{id}/pulse_results
Get Pulse results across all metrics · Retrieve Pulse Results (Beta)
#
Ask an LLM
“How are all the metrics trending for my test group against control?”
“Can I pull an experiment's full scorecard with a Bonferroni correction applied?”
Tell an agent
Get the full Pulse scorecard for experiment {id}, test group {test} vs control {control}.
Show all metric results for experiment {id} on {date}, {test} against {control}.
PUT
/console/v1/experiments/{id}/reset
Reset an experiment via the Console v1 API · Reset Experiment
#
Ask an LLM
“Which console v1 call wipes an experiment back to a fresh state?”
“Can I restart an experiment's data collection from zero with the v1 endpoint?”
Tell an agent destructive · confirm first
Reset experiment {id} through the console v1 endpoint.
Wipe v1 experiment {id} back to a fresh state and restart data collection.
POST
/console/v1/experiments/{id}/restart_as_new
Restart an experiment as a new copy · Restart As New Experiment
#
Ask an LLM
“Can I relaunch an experiment as a new one under a different name?”
“How do I rerun a test from scratch while keeping the original's results?”
Tell an agent
Restart experiment {id} as a new experiment named {name}.
Relaunch experiment {id} as a fresh copy called {name}.
GET
/console/v1/experiments/{id}/reviews
List review requests on an experiment · List Experiment Reviews
#
Ask an LLM
“What review requests are pending or done on my experiment?”
“Can I page through all approval requests filed against a test?”
Tell an agent
List the review requests on experiment {id}.
Show page {page} of reviews for experiment {id}, {limit} per page.
POST
/console/v1/experiments/{id}/reviews
Request a review of an experiment change · Create Experiment Review
#
Ask an LLM
“How do I ask teammates to approve a change to my experiment?”
“Can I request a review to ship a winning group or change rollout percentage?”
Tell an agent
Open a {type} review on experiment {id} for reviewers {reviewer_ids}.
Request a {type} review on experiment {id} to ship group {winning_group_id}.
GET
/console/v1/experiments/{id}/reviews/{reviewID}
Get one experiment review · Get Experiment Review
#
Ask an LLM
“What is the status of a specific review request on my experiment?”
“Who has signed off on one particular experiment review?”
Tell an agent
Get review {reviewID} on experiment {id}.
Check the approval status of review {reviewID} for experiment {id}.
PUT
/console/v1/experiments/{id}/reviews/{reviewID}
Edit an experiment review request · Edit Experiment Review
#
Ask an LLM
“Can I add reviewers to a review request I already opened?”
“How do I change the description of an existing experiment review?”
Tell an agent
Change the reviewers on review {reviewID} of experiment {id} to {reviewer_ids}.
Update the description of review {reviewID} on experiment {id} to {description}.
DELETE
/console/v1/experiments/{id}/reviews/{reviewID}
Cancel an experiment review request · Cancel Experiment Review
#
Ask an LLM
“How do I withdraw a review request I no longer need?”
“Can I cancel a pending approval on an experiment change?”
Tell an agent destructive · confirm first
Cancel review {reviewID} on experiment {id}.
Withdraw pending review request {reviewID} for experiment {id}.
PUT
/console/v1/experiments/{id}/reviews/{reviewID}/approve
Approve an experiment review · Approve Experiment Review
#
Ask an LLM
“How do I sign off on a teammate's experiment change request?”
“Can I leave a note when approving a review?”
Tell an agent
Approve review {reviewID} on experiment {id}.
Sign off on review {reviewID} for experiment {id} with the note {reason}.
PUT
/console/v1/experiments/{id}/reviews/{reviewID}/commit
Commit an approved experiment review · Commit Experiment Review
#
Ask an LLM
“Once a review is approved, how do I apply the change it proposed?”
“What makes an approved experiment change actually take effect?”
Tell an agent
Commit approved review {reviewID} on experiment {id}.
Apply the changes from approved review {reviewID} to experiment {id}.
PUT
/console/v1/experiments/{id}/reviews/{reviewID}/reject
Reject an experiment review · Reject Experiment Review
#
Ask an LLM
“How do I turn down a proposed change to an experiment?”
“Is a reason required when rejecting a review request?”
Tell an agent
Reject review {reviewID} on experiment {id} because {reason}.
Decline the change in review {reviewID} for experiment {id}, explaining {reason}.
POST
/console/v1/experiments/{id}/schedule_start
Schedule an experiment to start later · Schedule Experiment Start
#
Ask an LLM
“Can I set an experiment to go live automatically at a future time?”
“How do I schedule a test to launch next Monday morning?”
Tell an agent
Schedule experiment {id} to start at {scheduledTime}.
Set experiment {id} to launch automatically at {scheduledTime}.
PUT
/console/v1/experiments/{id}/start
Start an experiment with an analysis window · Start Experiment
#
Ask an LLM
“Can I start an experiment now and fix the start and end of its analysis window?”
“Is turbo mode something I can switch on at the moment a test starts?”
Tell an agent
Start v1 experiment {id} analysing from {analysisStartTime} to {analysisEndTime}.
Launch experiment {id} now with turbo mode {turboMode}.
GET
/console/v1/experiments/{id}/summary_charts
Get summary charts for an experiment · Retrieve Experiment Summary Charts (Beta)
#
Ask an LLM
“Can I get chart data summarising how an experiment is performing?”
“What summary charts exist for a test group compared to control?”
Tell an agent
Get the summary charts for experiment {id}.
Fetch summary chart data for experiment {id}, group {test} against {control}.
Ask an LLM
“How do I bring an archived experiment back?”
“Can an experiment be restored from the archive?”
Tell an agent
Unarchive experiment {id}.
Restore experiment {id} from the archive.
GET
/console/v1/experiments/{id}/versions
List an experiment's version history · List Experiment Versions
#
Ask an LLM
“Who changed my experiment's configuration and when?”
“Can I see previous versions of an experiment within a date range?”
Tell an agent
List the versions of experiment {id}.
Show version history for experiment {id} between {startDate} and {endDate}.
POST
/console/v1/experiments/assignment_source/{name}
Replace a named assignment source · Post Assignment Source
#
Ask an LLM
“How do I overwrite the SQL and column mapping of an existing assignment source?”
“Can I fully redefine an assignment source I already have by its name?”
Tell an agent
Replace assignment source {name} with SQL {sql}, timestamp column {timestampColumn} and experiment column {experimentIDColumn}.
Redefine existing assignment source {name} using group column {groupIDColumn} and ID mapping {idTypeMapping}.
PATCH
/console/v1/experiments/assignment_source/{name}
Update fields on an assignment source · Patch Assignment Source
#
Ask an LLM
“Can I change just the description or owner of an assignment source?”
“How do I adjust the reload hour on one assignment source?”
Tell an agent
Change the owner of assignment source {name} to {owner}.
Set the scheduled reload hour of assignment source {name} to {scheduledReloadHour}.
DELETE
/console/v1/experiments/assignment_source/{name}
Delete an assignment source · Delete Assignment Source
#
Ask an LLM
“How do I remove an assignment source I no longer use?”
“Can I delete a warehouse assignment source by name?”
Tell an agent destructive · confirm first
Delete assignment source {name}.
Remove the warehouse assignment source called {name}.
Ask an LLM
“Which warehouse tables are set up as assignment sources for experiments?”
“What assignment sources does my project have?”
Tell an agent
List all assignment sources.
Show page {page} of assignment sources, {limit} per page.
POST
/console/v1/experiments/assignment_sources
Create an assignment source · Create Assignment Source
#
Ask an LLM
“How do I register a new warehouse query that records which group each user got?”
“Can I dry-run a new assignment source before saving it?”
Tell an agent
Create assignment source {name} from SQL {sql} with timestamp column {timestampColumn}.
Add new assignment source {name} reading experiment IDs from {experimentIDColumn} and groups from {groupIDColumn}.
GET
/console/v1/experiments/entity_properties
List entity property sources · List Entity Property Sources
#
Ask an LLM
“Which entity property sources feed user attributes into my experiments?”
“Can I page through all the entity property sources in the project?”
Tell an agent
List all entity property sources.
Show page {page} of entity property sources with {limit} per page.
POST
/console/v1/experiments/entity_properties
Create an entity property source · Create Entity Property Source
#
Ask an LLM
“How do I add a warehouse query that supplies user properties for experiment analysis?”
“Can a new entity property source treat its timestamp as a day?”
Tell an agent
Create entity property source {name} from SQL {sql} with ID mapping {idTypeMapping}.
Add a new entity property source {name}, dry run {dryRun}.
GET
/console/v1/experiments/entity_property/{name}
Get an entity property source · Get Entity Property Source
#
Ask an LLM
“What SQL and ID mapping does a particular entity property source use?”
“Can I look up one entity property source by name?”
Tell an agent
Get entity property source {name}.
Show the SQL and ID mapping behind entity property source {name}.
POST
/console/v1/experiments/entity_property/{name}
Replace an entity property source · Post Entity Property Source
#
Ask an LLM
“How do I overwrite the query behind an existing entity property source?”
“Can I fully redefine a named entity property source in one call?”
Tell an agent
Replace entity property source {name} with SQL {sql} and ID mapping {idTypeMapping}.
Overwrite existing entity property source {name}'s whole definition using query {sql} and mapping {idTypeMapping}.
PATCH
/console/v1/experiments/entity_property/{name}
Update fields on an entity property source · Patch Entity Property Source
#
Ask an LLM
“Can I change just the description or tags of an entity property source?”
“How do I make an entity property source read-only?”
Tell an agent
Update the description of entity property source {name} to {description}.
Set entity property source {name} read-only to {isReadOnly}.
DELETE
/console/v1/experiments/entity_property/{name}
Delete an entity property source · Delete Entity Property Source
#
Ask an LLM
“How do I remove an entity property source from the project?”
“Can I delete an entity property source that is no longer needed?”
Tell an agent destructive · confirm first
Delete entity property source {name}.
Remove the entity property source called {name}.