Evals · JSON Structure

Evals Eval Case Structure

A single test case in an eval suite: input, optional expected output, retrieval context, metadata.

Type: object Properties: 8 Required: 2
EvalsLLM EvaluationAI QualityBenchmarksLLM as a JudgeObservabilityAgent EvaluationRAG EvaluationTest-Driven AI

EvalCase is a JSON Structure definition published by Evals, describing 8 properties, of which 2 are required. It conforms to the https://json-structure.org/meta/core/v0/# meta-schema.

Properties

id input input_structured expected expected_structured context metadata tags

Meta-schema: https://json-structure.org/meta/core/v0/#

JSON Structure

evals-eval-case-structure.json Raw ↑ 
{
  "$schema": "https://json-structure.org/meta/core/v0/#",
  "$id": "https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-structure/evals-eval-case-structure.json",
  "name": "EvalCase",
  "description": "A single test case in an eval suite: input, optional expected output, retrieval context, metadata.",
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "input": { "type": "string" },
    "input_structured": { "type": "object" },
    "expected": { "type": "string" },
    "expected_structured": { "type": "object" },
    "context": {
      "type": "array",
      "items": { "type": "string" }
    },
    "metadata": {
      "type": "object",
      "properties": {
        "domain": { "type": "string" },
        "difficulty": {
          "type": "string",
          "enum": ["easy", "medium", "hard"]
        },
        "persona": { "type": "string" },
        "source": { "type": "string", "format": "uri" }
      }
    },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["id", "input"]
}