SuccessEvaluationPlan

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "#/components/schemas/SuccessEvaluationPlan",
  "title": "SuccessEvaluationPlan",
  "type": "object",
  "properties": {
    "rubric": {
      "type": "string",
      "enum": [
        "NumericScale",
        "DescriptiveScale",
        "Checklist",
        "Matrix",
        "PercentageScale",
        "LikertScale",
        "AutomaticRubric",
        "PassFail"
      ],
      "description": "This enforces the rubric of the evaluation. The output is stored in `call.analysis.successEvaluation`.\n\nOptions include:\n- 'NumericScale': A scale of 1 to 10.\n- 'DescriptiveScale': A scale of Excellent, Good, Fair, Poor.\n- 'Checklist': A checklist of criteria and their status.\n- 'Matrix': A grid that evaluates multiple criteria across different performance levels.\n- 'PercentageScale': A scale of 0% to 100%.\n- 'LikertScale': A scale of Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree.\n- 'AutomaticRubric': Automatically break down evaluation into several criteria, each with its own score.\n- 'PassFail': A simple 'true' if call passed, 'false' if not.\n\nDefault is 'PassFail'."
    },
    "messages": {
      "description": "These are the messages used to generate the success evaluation.\n\n@default: ```\n[\n  {\n    \"role\": \"system\",\n    \"content\": \"You are an expert call evaluator. You will be given a transcript of a call and the system prompt of the AI participant. Determine if the call was successful based on the objectives inferred from the system prompt. DO NOT return anything except the result.\\n\\nRubric:\\\\n{{rubric}}\\n\\nOnly respond with the result.\"\n  },\n  {\n    \"role\": \"user\",\n    \"content\": \"Here is the transcript:\\n\\n{{transcript}}\\n\\n\"\n  },\n  {\n    \"role\": \"user\",\n    \"content\": \"Here was the system prompt of the call:\\n\\n{{systemPrompt}}\\n\\n. Here is the ended reason of the call:\\n\\n{{endedReason}}\\n\\n\"\n  }\n]```\n\nYou can customize by providing any messages you want.\n\nHere are the template variables available:\n- {{transcript}}: the transcript of the call from `call.artifact.transcript`- {{systemPrompt}}: the system prompt of the call from `assistant.model.messages[type=system].content`- {{messages}}: the messages of the call from `assistant.model.messages`- {{rubric}}: the rubric of the success evaluation from `successEvaluationPlan.rubric`- {{endedReason}}: the ended reason of the call from `call.endedReason`",
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "enabled": {
      "type": "boolean",
      "description": "This determines whether a success evaluation is generated and stored in `call.analysis.successEvaluation`. Defaults to true.\n\nUsage:\n- If you want to disable the success evaluation, set this to false.\n\n@default true"
    },
    "timeoutSeconds": {
      "type": "number",
      "description": "This is how long the request is tried before giving up. When request times out, `call.analysis.successEvaluation` will be empty.\n\nUsage:\n- To guarantee the success evaluation is generated, set this value high. Note, this will delay the end of call report in cases where model is slow to respond.\n\n@default 5 seconds",
      "minimum": 1,
      "maximum": 60
    }
  }
}