Evals · JSON Structure

Evals Eval Suite Structure

A named collection of eval cases plus scorers and aggregation policy.

Type: object Properties: 10 Required: 4
EvalsLLM EvaluationAI QualityBenchmarksLLM as a JudgeObservabilityAgent EvaluationRAG EvaluationTest-Driven AI

EvalSuite is a JSON Structure definition published by Evals, describing 10 properties, of which 4 are required. It conforms to the https://json-structure.org/meta/core/v0/# meta-schema.

Properties

id name description version dataset_id scorers policy tags created modified

Meta-schema: https://json-structure.org/meta/core/v0/#

JSON Structure

evals-eval-suite-structure.json Raw ↑ 
{
  "$schema": "https://json-structure.org/meta/core/v0/#",
  "$id": "https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-structure/evals-eval-suite-structure.json",
  "name": "EvalSuite",
  "description": "A named collection of eval cases plus scorers and aggregation policy.",
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "name": { "type": "string" },
    "description": { "type": "string" },
    "version": { "type": "string" },
    "dataset_id": { "type": "string" },
    "scorers": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": { "type": "string" },
          "name": { "type": "string" },
          "type": {
            "type": "string",
            "enum": [
              "code",
              "llm_judge",
              "human",
              "heuristic",
              "reference_based",
              "reference_free",
              "pairwise"
            ]
          },
          "threshold": { "type": "number" }
        }
      }
    },
    "policy": {
      "type": "object",
      "properties": {
        "aggregation": {
          "type": "string",
          "enum": ["mean", "median", "pass_rate", "min", "max"]
        },
        "fail_on_threshold": { "type": "boolean" }
      }
    },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    },
    "created": { "type": "string", "format": "date-time" },
    "modified": { "type": "string", "format": "date-time" }
  },
  "required": ["id", "name", "dataset_id", "scorers"]
}