Connic · Schema

ABTestResponse

An A/B test that routes part of an agent's traffic to a variant, with run totals per arm.

CompanyArtificial IntelligencePlatformMunichAutomation

Properties

Name	Type	Description
id	string
project_id	string	Project the test belongs to.
environment_id	string	Environment the test runs in.
agent_name	string	Base (control) agent.
test_name	string	Variant suffix.
variant_agent_name	string	Variant agent name, `{agent_name}-test-{test_name}`.
traffic_percentage	integer	Percentage of the agent's traffic routed to the variant.
status	object	Lifecycle state. Unless the test is running, all traffic goes to the base agent.
analysis_mode	object	How results are analyzed. Fixed once the test leaves draft.
analysis_plan	object	Statistical plan of a confidence test. null for exploratory tests.
min_sample_size	integer	Completed runs needed in each group before the comparison reports enough data.
auto_rollback_enabled	boolean	Whether the failure guardrail is on.
rollback_threshold	number	Variant failure rate (0 to 1) above which the failure guardrail pauses the test.
rollback_window	integer	Number of most recent finished variant runs the failure guardrail evaluates.
quality_guardrail_enabled	boolean	Whether the quality guardrail is on.
quality_judge_id	object	Judge for the quality guardrail. null when unset or without `judges.view`.
quality_threshold	number	Lowest acceptable average normalized score (0 to 1) for the variant.
quality_window	integer	Number of most recent evaluated variant runs the quality guardrail averages.
quality_min_evaluations	integer	Evaluations needed before the quality guardrail can act.
winner	object	Manually chosen winner of an exploratory test. null when none.
recommendation	object	Verdict of a confidence test, or `safety_stopped` for any test stopped automatically.
recommendation_reason	object	Explanation of `recommendation`.
error	object	Why the test was stopped automatically. Cleared when the test resumes.
stop_reason	object	Why the test last stopped. null while running or before the first stop.
stop_details	object	Details for `stop_reason`, such as the observed failure rate and window for `failure_guardrail`.
created_at	string
updated_at	string
started_at	object	When the test first started. null for drafts.
stopped_at	object	When the test was last paused or stopped. Cleared when it resumes.
concluded_at	object	When the test was concluded.
control_runs	integer	Completed control runs. Confidence tests exclude forced runs.
variant_runs	integer	Completed variant runs. Confidence tests exclude forced runs.
control_avg_cost	object	Always null. Cost statistics are returned by the comparison endpoint.
variant_avg_cost	object	Always null. Cost statistics are returned by the comparison endpoint.
control_avg_duration	object	Average duration of completed control runs, in milliseconds.
variant_avg_duration	object	Average duration of completed variant runs, in milliseconds.
control_avg_score	object	Average normalized judge score (0 to 1) of control runs, from the metric or quality judge when set. null without evaluations.
variant_avg_score	object	Average normalized judge score (0 to 1) of variant runs, from the metric or quality judge when set. null without evaluations.

View JSON Schema on GitHub

JSON Schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://artifacts.apis.io/connic-co/json-schema/connic-co-abtest-response-schema.json",
  "title": "ABTestResponse",
  "description": "An A/B test that routes part of an agent's traffic to a variant, with run totals per arm.",
  "x-generated": "2026-11-04",
  "x-method": "derived",
  "x-generator": "derive-json-schema.py",
  "x-source": "openapi/_original/connic-co-public-openapi.json#/components/schemas/ABTestResponse",
  "properties": {
    "id": {
      "type": "string",
      "format": "uuid",
      "title": "Id"
    },
    "project_id": {
      "type": "string",
      "format": "uuid",
      "title": "Project Id",
      "description": "Project the test belongs to."
    },
    "environment_id": {
      "type": "string",
      "format": "uuid",
      "title": "Environment Id",
      "description": "Environment the test runs in."
    },
    "agent_name": {
      "type": "string",
      "title": "Agent Name",
      "description": "Base (control) agent."
    },
    "test_name": {
      "type": "string",
      "title": "Test Name",
      "description": "Variant suffix."
    },
    "variant_agent_name": {
      "type": "string",
      "title": "Variant Agent Name",
      "description": "Variant agent name, `{agent_name}-test-{test_name}`."
    },
    "traffic_percentage": {
      "type": "integer",
      "title": "Traffic Percentage",
      "description": "Percentage of the agent's traffic routed to the variant."
    },
    "status": {
      "$ref": "#/$defs/ABTestStatus",
      "description": "Lifecycle state. Unless the test is running, all traffic goes to the base agent."
    },
    "analysis_mode": {
      "$ref": "#/$defs/ABTestAnalysisMode",
      "description": "How results are analyzed. Fixed once the test leaves draft.",
      "default": "exploratory"
    },
    "analysis_plan": {
      "anyOf": [
        {
          "$ref": "#/$defs/ABTestAnalysisPlanResponse"
        },
        {
          "type": "null"
        }
      ],
      "description": "Statistical plan of a confidence test. null for exploratory tests."
    },
    "min_sample_size": {
      "type": "integer",
      "title": "Min Sample Size",
      "description": "Completed runs needed in each group before the comparison reports enough data."
    },
    "auto_rollback_enabled": {
      "type": "boolean",
      "title": "Auto Rollback Enabled",
      "description": "Whether the failure guardrail is on."
    },
    "rollback_threshold": {
      "type": "number",
      "title": "Rollback Threshold",
      "description": "Variant failure rate (0 to 1) above which the failure guardrail pauses the test."
    },
    "rollback_window": {
      "type": "integer",
      "title": "Rollback Window",
      "description": "Number of most recent finished variant runs the failure guardrail evaluates."
    },
    "quality_guardrail_enabled": {
      "type": "boolean",
      "title": "Quality Guardrail Enabled",
      "description": "Whether the quality guardrail is on.",
      "default": false
    },
    "quality_judge_id": {
      "anyOf": [
        {
          "type": "string",
          "format": "uuid"
        },
        {
          "type": "null"
        }
      ],
      "title": "Quality Judge Id",
      "description": "Judge for the quality guardrail. null when unset or without `judges.view`."
    },
    "quality_threshold": {
      "type": "number",
      "title": "Quality Threshold",
      "description": "Lowest acceptable average normalized score (0 to 1) for the variant.",
      "default": 0.0
    },
    "quality_window": {
      "type": "integer",
      "title": "Quality Window",
      "description": "Number of most recent evaluated variant runs the quality guardrail averages.",
      "default": 20
    },
    "quality_min_evaluations": {
      "type": "integer",
      "title": "Quality Min Evaluations",
      "description": "Evaluations needed before the quality guardrail can act.",
      "default": 10
    },
    "winner": {
      "anyOf": [
        {
          "type": "string",
          "enum": [
            "control",
            "variant"
          ]
        },
        {
          "type": "null"
        }
      ],
      "title": "Winner",
      "description": "Manually chosen winner of an exploratory test. null when none."
    },
    "recommendation": {
      "anyOf": [
        {
          "$ref": "#/$defs/ABTestRecommendation"
        },
        {
          "type": "null"
        }
      ],
      "description": "Verdict of a confidence test, or `safety_stopped` for any test stopped automatically."
    },
    "recommendation_reason": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "title": "Recommendation Reason",
      "description": "Explanation of `recommendation`."
    },
    "error": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "title": "Error",
      "description": "Why the test was stopped automatically. Cleared when the test resumes."
    },
    "stop_reason": {
      "anyOf": [
        {
          "$ref": "#/$defs/ABTestStopReason"
        },
        {
          "type": "null"
        }
      ],
      "description": "Why the test last stopped. null while running or before the first stop."
    },
    "stop_details": {
      "additionalProperties": true,
      "type": "object",
      "title": "Stop Details",
      "description": "Details for `stop_reason`, such as the observed failure rate and window for `failure_guardrail`."
    },
    "created_at": {
      "type": "string",
      "format": "date-time",
      "title": "Created At"
    },
    "updated_at": {
      "type": "string",
      "format": "date-time",
      "title": "Updated At"
    },
    "started_at": {
      "anyOf": [
        {
          "type": "string",
          "format": "date-time"
        },
        {
          "type": "null"
        }
      ],
      "title": "Started At",
      "description": "When the test first started. null for drafts."
    },
    "stopped_at": {
      "anyOf": [
        {
          "type": "string",
          "format": "date-time"
        },
        {
          "type": "null"
        }
      ],
      "title": "Stopped At",
      "description": "When the test was last paused or stopped. Cleared when it resumes."
    },
    "concluded_at": {
      "anyOf": [
        {
          "type": "string",
          "format": "date-time"
        },
        {
          "type": "null"
        }
      ],
      "title": "Concluded At",
      "description": "When the test was concluded."
    },
    "control_runs": {
      "type": "integer",
      "title": "Control Runs",
      "description": "Completed control runs. Confidence tests exclude forced runs.",
      "default": 0
    },
    "variant_runs": {
      "type": "integer",
      "title": "Variant Runs",
      "description": "Completed variant runs. Confidence tests exclude forced runs.",
      "default": 0
    },
    "control_avg_cost": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "title": "Control Avg Cost",
      "description": "Always null. Cost statistics are returned by the comparison endpoint."
    },
    "variant_avg_cost": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "title": "Variant Avg Cost",
      "description": "Always null. Cost statistics are returned by the comparison endpoint."
    },
    "control_avg_duration": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "title": "Control Avg Duration",
      "description": "Average duration of completed control runs, in milliseconds."
    },
    "variant_avg_duration": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "title": "Variant Avg Duration",
      "description": "Average duration of completed variant runs, in milliseconds."
    },
    "control_avg_score": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "title": "Control Avg Score",
      "description": "Average normalized judge score (0 to 1) of control runs, from the metric or quality judge when set. null without evaluations."
    },
    "variant_avg_score": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "title": "Variant Avg Score",
      "description": "Average normalized judge score (0 to 1) of variant runs, from the metric or quality judge when set. null without evaluations."
    }
  },
  "type": "object",
  "required": [
    "id",
    "project_id",
    "environment_id",
    "agent_name",
    "test_name",
    "variant_agent_name",
    "traffic_percentage",
    "status",
    "min_sample_size",
    "auto_rollback_enabled",
    "rollback_threshold",
    "rollback_window",
    "created_at",
    "updated_at"
  ],
  "$defs": {
    "ABTestAnalysisMode": {
      "type": "string",
      "enum": [
        "exploratory",
        "confidence"
      ],
      "title": "ABTestAnalysisMode",
      "description": "`exploratory` compares descriptive metrics; `confidence` follows a locked statistical plan with planned decision looks."
    },
    "ABTestAnalysisPlanResponse": {
      "properties": {
        "primary_metric": {
          "type": "string",
          "enum": [
            "success_rate",
            "judge_score"
          ],
          "title": "Primary Metric",
          "description": "`success_rate` compares the share of completed runs; `judge_score` compares the normalized score from `judge_id`."
        },
        "objective": {
          "$ref": "#/$defs/ABTestObjective",
          "description": "Hypothesis to test. `noninferiority` requires `noninferiority_margin`.",
          "default": "superiority"
        },
        "direction": {
          "$ref": "#/$defs/ABTestMetricDirection",
          "description": "Whether a higher or lower metric value is better.",
          "default": "increase"
        },
        "baseline_value": {
          "type": "number",
          "title": "Baseline Value",
          "description": "Expected control value: a success rate between 0 and 1 (exclusive), or a judge score from 0 to 1."
        },
        "baseline_stddev": {
          "anyOf": [
            {
              "type": "number",
              "minimum": 0.0
            },
            {
              "type": "null"
            }
          ],
          "title": "Baseline Stddev",
          "description": "Expected standard deviation of the normalized judge score. Required and positive for `judge_score`."
        },
        "minimum_detectable_effect": {
          "type": "number",
          "exclusiveMinimum": 0.0,
          "title": "Minimum Detectable Effect",
          "description": "Smallest absolute difference worth detecting, for example 0.05 for 5 percentage points."
        },
        "noninferiority_margin": {
          "anyOf": [
            {
              "type": "number",
              "exclusiveMinimum": 0.0
            },
            {
              "type": "null"
            }
          ],
          "title": "Noninferiority Margin",
          "description": "Largest acceptable absolute shortfall versus control. Required for `noninferiority`."
        },
        "alpha": {
          "type": "number",
          "exclusiveMaximum": 0.5,
          "exclusiveMinimum": 0.0,
          "title": "Alpha",
          "description": "Two-sided significance level.",
          "default": 0.05
        },
        "power": {
          "type": "number",
          "exclusiveMaximum": 1.0,
          "exclusiveMinimum": 0.5,
          "title": "Power",
          "description": "Probability of detecting the planned effect.",
          "default": 0.8
        },
        "randomization_unit": {
          "$ref": "#/$defs/ABTestRandomizationUnit",
          "description": "Whether each run or each session is assigned to an arm.",
          "default": "run"
        },
        "judge_id": {
          "anyOf": [
            {
              "type": "string",
              "format": "uuid"
            },
            {
              "type": "null"
            }
          ],
          "title": "Judge Id",
          "description": "Judge that scores runs for `judge_score`. It must belong to the same agent and environment, be enabled and automatic, evaluate every run and have no filter. Must be null for `success_rate`."
        },
        "futility_threshold": {
          "type": "number",
          "exclusiveMaximum": 1.0,
          "exclusiveMinimum": 0.0,
          "title": "Futility Threshold",
          "description": "Conditional power below which a look is flagged as unlikely to succeed. Advisory only; it does not stop the test.",
          "default": 0.2
        },
        "id": {
          "type": "string",
          "format": "uuid",
          "title": "Id"
        },
        "ab_test_id": {
          "type": "string",
          "format": "uuid",
          "title": "Ab Test Id",
          "description": "A/B test the plan belongs to."
        },
        "control_allocation": {
          "type": "number",
          "title": "Control Allocation",
          "description": "Share of assigned traffic that goes to control."
        },
        "variant_allocation": {
          "type": "number",
          "title": "Variant Allocation",
          "description": "Share of assigned traffic that goes to the variant."
        },
        "judge_snapshot": {
          "additionalProperties": true,
          "type": "object",
          "title": "Judge Snapshot",
          "description": "Metric judge configuration captured at the first start. Empty for `success_rate` plans or before the start; only `redacted: true` without `judges.view`."
        },
        "quality_judge_snapshot": {
          "additionalProperties": true,
          "type": "object",
          "title": "Quality Judge Snapshot",
          "description": "Quality guardrail judge configuration captured at the first start. Empty when the guardrail is off; only `redacted: true` without `judges.view`."
        },
        "planned_control_samples": {
          "type": "integer",
          "title": "Planned Control Samples",
          "description": "Maximum number of control units the plan collects."
        },
        "planned_variant_samples": {
          "type": "integer",
          "title": "Planned Variant Samples",
          "description": "Maximum number of variant units the plan collects."
        },
        "look_fractions": {
          "items": {
            "type": "number"
          },
          "type": "array",
          "title": "Look Fractions",
          "description": "Shares of the planned sample at which analysis looks happen."
        },
        "boundaries": {
          "items": {
            "additionalProperties": true,
            "type": "object"
          },
          "type": "array",
          "title": "Boundaries",
          "description": "Decision boundary for each planned look, in the same shape as the plan preview `boundaries`."
        },
        "design_version": {
          "type": "string",
          "title": "Design Version",
          "description": "Identifier of the sequential design, such as `two-sided-obf-v1`."
        },
        "srm_threshold": {
          "type": "number",
          "title": "Srm Threshold",
          "description": "p-value below which a sample-ratio mismatch between the planned and observed split stops the test."
        },
        "outcome_policy": {
          "type": "string",
          "title": "Outcome Policy",
          "description": "Identifier of the rule that turns each assigned unit into an outcome."
        },
        "locked_at": {
          "anyOf": [
            {
              "type": "string",
              "format": "date-time"
            },
            {
              "type": "null"
            }
          ],
          "title": "Locked At",
          "description": "When the plan was locked by the test's first start. null while the test is a draft."
        },
        "created_at": {
          "type": "string",
          "format": "date-time",
          "title": "Created At"
        },
        "updated_at": {
          "type": "string",
          "format": "date-time",
          "title": "Updated At"
        }
      },
      "type": "object",
      "required": [
        "primary_metric",
        "baseline_value",
        "minimum_detectable_effect",
        "id",
        "ab_test_id",
        "control_allocation",
        "variant_allocation",
        "planned_control_samples",
        "planned_variant_samples",
        "look_fractions",
        "boundaries",
        "design_version",
        "srm_threshold",
        "outcome_policy",
        "created_at",
        "updated_at"
      ],
      "title": "ABTestAnalysisPlanResponse",
      "description": "Statistical plan of a confidence-mode A/B test, locked when the test first starts."
    },
    "ABTestMetricDirection": {
      "type": "string",
      "enum": [
        "increase",
        "decrease"
      ],
      "title": "ABTestMetricDirection",
      "description": "Which change in the primary metric counts as better: `increase` or `decrease`."
    },
    "ABTestObjective": {
      "type": "string",
      "enum": [
        "superiority",
        "noninferiority"
      ],
      "title": "ABTestObjective",
      "description": "`superiority` tests whether the variant beats control; `noninferiority` tests that it is not worse by more than a margin."
    },
    "ABTestRandomizationUnit": {
      "type": "string",
      "enum": [
        "run",
        "session"
      ],
      "title": "ABTestRandomizationUnit",
      "description": "`run` assigns each run independently; `session` keeps all runs of one session in the same arm."
    },
    "ABTestRecommendation": {
      "type": "string",
      "enum": [
        "collecting",
        "continue",
        "variant_recommended",
        "control_recommended",
        "inconclusive",
        "invalid_srm",
        "safety_stopped"
      ],
      "title": "ABTestRecommendation",
      "description": "Test verdict: `collecting` before the first planned look, `continue` after a look without a decision, `variant_recommended` or `control_recommended` when a boundary was crossed, `inconclusive`, `invalid_srm` after a sample-ratio mismatch, or `safety_stopped` when the test was stopped automatically."
    },
    "ABTestStatus": {
      "type": "string",
      "enum": [
        "draft",
        "running",
        "paused",
        "concluded",
        "rolled_back",
        "failed"
      ],
      "title": "ABTestStatus",
      "description": "A/B test state: `draft` until started, `running` while routing traffic, `paused`, `concluded`, or `failed` when the variant is missing from the active deployment. `rolled_back` is no longer set."
    },
    "ABTestStopReason": {
      "type": "string",
      "enum": [
        "manual",
        "failure_guardrail",
        "quality_guardrail",
        "efficacy",
        "futility",
        "srm",
        "analysis_error",
        "variant_unavailable"
      ],
      "title": "ABTestStopReason",
      "description": "Why a test stopped: `manual`, a guardrail (`failure_guardrail`, `quality_guardrail`), a planned look (`efficacy` when a boundary was crossed, `futility` when the final look was inconclusive), `srm` (sample-ratio mismatch), `analysis_error` (metric judge evaluations failed), or `variant_unavailable`."
    }
  }
}

Work with this as data

Every JSON Schema here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for schemas

4 MCP tools reach this

find_json_schemasBrowse and filter every JSON Schema in the catalog.
apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
resolveTurn a domain, URL or GitHub org into the provider it belongs to.
find_cohortsEvery scored population of providers in the catalog.

All 92 tools →

Call it yourself

curl for this page

This JSON Schema

curl "https://apis.io/api/v1/json-schemas/connic-co-abtest-response"

All schemas

curl "https://apis.io/api/v1/json-schemas?limit=25"

Calls without X-API-Key need no key. Calls with it need a paid plan — get a key.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.