Evals · JSON Structure

Evals Dataset Structure

A collection of EvalCases plus provenance, license, splits, and task metadata.

Type: object Properties: 12 Required: 3

EvalsLLM EvaluationAI QualityBenchmarksLLM as a JudgeObservabilityAgent EvaluationRAG EvaluationTest-Driven AI

EvalDataset is a JSON Structure definition published by Evals, describing 12 properties, of which 3 are required. It conforms to the https://json-structure.org/meta/core/v0/# meta-schema.

Properties

id name description version task source license splits case_count tags created modified

Meta-schema: https://json-structure.org/meta/core/v0/#

JSON Structure

{
  "$schema": "https://json-structure.org/meta/core/v0/#",
  "$id": "https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-structure/evals-dataset-structure.json",
  "name": "EvalDataset",
  "description": "A collection of EvalCases plus provenance, license, splits, and task metadata.",
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "name": { "type": "string" },
    "description": { "type": "string" },
    "version": { "type": "string" },
    "task": {
      "type": "string",
      "enum": [
        "qa",
        "rag",
        "code_generation",
        "summarization",
        "classification",
        "agent_task",
        "safety",
        "multi_turn_dialogue",
        "knowledge",
        "reasoning"
      ]
    },
    "source": { "type": "string", "format": "uri" },
    "license": { "type": "string" },
    "splits": {
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "count": { "type": "integer" },
          "uri": { "type": "string", "format": "uri" }
        }
      }
    },
    "case_count": { "type": "integer" },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    },
    "created": { "type": "string", "format": "date-time" },
    "modified": { "type": "string", "format": "date-time" }
  },
  "required": ["id", "name", "task"]
}