Evals · JSON Structure

Evals Judge Structure

A specialized LLM-as-a-judge (or human) scorer with model, prompt, rubric, and calibration data.

Type: object Properties: 10 Required: 3

EvalsLLM EvaluationAI QualityBenchmarksLLM as a JudgeObservabilityAgent EvaluationRAG EvaluationTest-Driven AI

Judge is a JSON Structure definition published by Evals, describing 10 properties, of which 3 are required. It conforms to the https://json-structure.org/meta/core/v0/# meta-schema.

Properties

id name description judge_kind model prompt_template rubric output_format calibration tags

Meta-schema: https://json-structure.org/meta/core/v0/#

JSON Structure

{
  "$schema": "https://json-structure.org/meta/core/v0/#",
  "$id": "https://raw.githubusercontent.com/api-evangelist/evals/refs/heads/main/json-structure/evals-judge-structure.json",
  "name": "Judge",
  "description": "A specialized LLM-as-a-judge (or human) scorer with model, prompt, rubric, and calibration data.",
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "name": { "type": "string" },
    "description": { "type": "string" },
    "judge_kind": {
      "type": "string",
      "enum": ["llm", "human", "model_panel", "human_panel", "distilled_evaluator"]
    },
    "model": {
      "type": "object",
      "properties": {
        "provider": { "type": "string" },
        "name": { "type": "string" },
        "version": { "type": "string" }
      }
    },
    "prompt_template": { "type": "string" },
    "rubric": { "type": "string" },
    "output_format": {
      "type": "string",
      "enum": ["score", "score_and_rationale", "label", "label_and_rationale", "pairwise_preference"]
    },
    "calibration": {
      "type": "object",
      "properties": {
        "human_agreement": { "type": "number" },
        "kappa": { "type": "number" },
        "sample_size": { "type": "integer" },
        "calibrated_on": { "type": "string", "format": "date-time" }
      }
    },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["id", "name", "judge_kind"]
}