Scorecard Testsets API
The Testsets API from Scorecard — 2 operation(s) for testsets.
OpenAPI Specification
openapi: 3.1.0
info:
title: Scorecard Metrics Testsets API
description: REST API for Scorecard
version: 1.0.0
servers:
- url: https://api2.scorecard.io/api/v2
security:
- ApiKeyAuth: []
tags:
- name: Testsets
paths:
/testsets/{testsetId}:
get:
operationId: getTestset
summary: Get Testset
parameters:
- in: path
name: testsetId
description: The ID of the Testset.
schema:
type: string
example: '246'
required: true
responses:
'200':
description: Successfully retrieved Testset
content:
application/json:
schema:
$ref: '#/components/schemas/Testset'
examples:
Complete Testset details:
value:
id: '246'
name: Long Context Q&A
description: Testset for long context Q&A chatbot.
jsonSchema:
type: object
properties:
question:
type: string
idealAnswer:
type: string
provenance:
type: string
geo:
type: string
fieldMapping:
inputs:
- question
expected:
- idealAnswer
metadata:
- provenance
- geo
summary: Complete Testset details
description: Example response showing all details of a Testset including its schema definition and field mappings.
'401':
$ref: '#/components/responses/UnauthenticatedError'
'500':
$ref: '#/components/responses/ServiceError'
x-codeSamples:
- lang: JavaScript
source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testset = await client.testsets.get('246');\n\nconsole.log(testset.id);"
- lang: Python
source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestset = client.testsets.get(\n \"246\",\n)\nprint(testset.id)"
- lang: cURL
source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\""
tags:
- Testsets
patch:
operationId: updateTestset
summary: Update Testset
description: 'Update a Testset. Only the fields provided in the request body will be updated.
If a field is provided, the new content will replace the existing content.
If a field is not provided, the existing content will remain unchanged.
When updating the schema:
- If field mappings are not provided and existing mappings reference fields that no longer exist, those mappings will be automatically removed
- To preserve all existing mappings, ensure all referenced fields remain in the updated schema
- For complete control, provide both schema and fieldMapping when updating the schema'
parameters:
- in: path
name: testsetId
description: The ID of the Testset to update.
schema:
type: string
example: '246'
required: true
requestBody:
content:
application/json:
schema:
type: object
properties:
name:
type: string
description: The name of the Testset.
description:
type: string
description: The description of the Testset.
jsonSchema:
type: object
description: The JSON schema for each Testcase in the Testset.
additionalProperties: true
fieldMapping:
type: object
properties:
inputs:
type: array
items:
type: string
description: Fields that represent inputs to the AI system.
expected:
type: array
items:
type: string
description: Fields that represent expected outputs.
metadata:
type: array
items:
type: string
description: Fields that are not inputs or expected outputs.
required:
- inputs
- expected
- metadata
description: Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata.
examples:
Update metadata only:
value:
name: Updated Q&A Testset
description: Updated description for the Q&A Testset.
summary: Update metadata only
description: Simple metadata update without changing schema or mappings. Updates only the name and description fields while preserving the existing schema and field mappings.
Remove schema field:
value:
jsonSchema:
type: object
properties:
question:
type: string
idealAnswer:
type: string
provenance:
type: string
summary: Remove schema field
description: This request removes the 'geo' field that existed in the original schema, but doesn't explicitly update the field mappings. Scorecard will automatically remove any field mappings that reference deleted fields.
Full schema revision:
value:
jsonSchema:
type: object
properties:
question:
type: string
idealAnswer:
type: string
provenance:
type: string
fieldMapping:
inputs:
- question
expected:
- idealAnswer
metadata: []
summary: Full schema revision
description: Explicit update of both schema and field mappings, allowing complete control over the Testset structure. This example removes the 'geo' field and explicitly updates the field mappings to exclude 'provenance' from metadata.
responses:
'200':
description: Testset updated successfully
content:
application/json:
schema:
$ref: '#/components/schemas/Testset'
examples:
Updated metadata only:
value:
id: '246'
name: Updated Q&A Testset
description: Updated description for the Q&A Testset.
jsonSchema:
type: object
properties:
question:
type: string
idealAnswer:
type: string
provenance:
type: string
geo:
type: string
fieldMapping:
inputs:
- question
expected:
- idealAnswer
metadata:
- provenance
- geo
summary: Updated metadata only
description: Result after updating only the Testset's name and description. All schema fields and mappings remain unchanged.
Auto-updated mappings:
value:
id: '246'
name: Long Context Q&A
description: Testset for long context Q&A chatbot.
jsonSchema:
type: object
properties:
question:
type: string
idealAnswer:
type: string
provenance:
type: string
fieldMapping:
inputs:
- question
expected:
- idealAnswer
metadata:
- provenance
summary: Auto-updated mappings
description: Result after schema update with automatic field mapping cleanup. The 'geo' field has been automatically removed from the metadata mapping since it no longer exists in the schema.
Custom field mapping:
value:
id: '246'
name: Long Context Q&A
description: Testset for long context Q&A chatbot.
jsonSchema:
type: object
properties:
question:
type: string
idealAnswer:
type: string
provenance:
type: string
fieldMapping:
inputs:
- question
expected:
- idealAnswer
metadata: []
summary: Custom field mapping
description: Result after explicit schema and field mapping update. Note that 'provenance' is not included in metadata since it wasn't specified in the request's field mapping.
'401':
$ref: '#/components/responses/UnauthenticatedError'
'500':
$ref: '#/components/responses/ServiceError'
x-codeSamples:
- lang: JavaScript
source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testset = await client.testsets.update('246', {\n description: 'Updated description for the Q&A Testset.',\n name: 'Updated Q&A Testset',\n});\n\nconsole.log(testset.id);"
- lang: Python
source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestset = client.testsets.update(\n testset_id=\"246\",\n description=\"Updated description for the Q&A Testset.\",\n name=\"Updated Q&A Testset\",\n)\nprint(testset.id)"
- lang: cURL
source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID \\\n -X PATCH \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\""
tags:
- Testsets
delete:
operationId: deleteTestset
summary: Delete Testset
parameters:
- in: path
name: testsetId
description: The ID of the Testset to delete.
schema:
type: string
example: '246'
required: true
responses:
'200':
description: Testset deleted successfully
content:
application/json:
schema:
type: object
properties:
success:
type: boolean
description: Whether the deletion was successful.
required:
- success
'401':
$ref: '#/components/responses/UnauthenticatedError'
'500':
$ref: '#/components/responses/ServiceError'
x-codeSamples:
- lang: JavaScript
source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testset = await client.testsets.delete('246');\n\nconsole.log(testset.success);"
- lang: Python
source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestset = client.testsets.delete(\n \"246\",\n)\nprint(testset.success)"
- lang: cURL
source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID \\\n -X DELETE \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\""
tags:
- Testsets
/testsets/{testsetId}/testcases:
get:
operationId: listTestcases
summary: List Testcases in Testset
description: Retrieve a paginated list of Testcases belonging to a Testset.
parameters:
- in: path
name: testsetId
description: The ID of the Testset to list Testcases from.
schema:
type: string
example: '246'
required: true
- in: query
name: limit
description: Maximum number of items to return (1-100). Use with `cursor` for pagination through large sets.
schema:
type: integer
exclusiveMinimum: 0
default: 20
example: 20
- in: query
name: cursor
description: Cursor for pagination. Pass the `nextCursor` from the previous response to get the next page of results.
schema:
type: string
example: '123'
responses:
'200':
description: Successfully retrieved list of Testcases.
content:
application/json:
schema:
type: object
properties:
data:
type: array
items:
$ref: '#/components/schemas/Testcase'
nextCursor:
type:
- string
- 'null'
hasMore:
type: boolean
total:
type: integer
minimum: 0
required:
- data
- nextCursor
- hasMore
examples:
Testcase list with pagination:
value:
data:
- id: '123'
testsetId: '246'
jsonData:
question: What is the capital of France?
idealAnswer: Paris
provenance: hand_curated
inputs:
question: What is the capital of France?
expected:
idealAnswer: Paris
- id: '124'
testsetId: '246'
jsonData:
question: What is the largest planet in our solar system?
idealAnswer: Jupiter
provenance: synthetic
inputs:
question: What is the largest planet in our solar system?
expected:
idealAnswer: Jupiter
- id: '125'
testsetId: '246'
jsonData:
question: What is the deepest ocean on Earth?
provenance: user_feedback
inputs:
question: What is the deepest ocean on Earth?
expected: {}
validationErrors:
- path: /data
message: Required field 'idealAnswer' is missing
nextCursor: '126'
hasMore: true
summary: Testcase list with pagination
description: Example response showing a paginated list of Testcases.
'401':
$ref: '#/components/responses/UnauthenticatedError'
'500':
$ref: '#/components/responses/ServiceError'
x-codeSamples:
- lang: JavaScript
source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\n// Automatically fetches more pages as needed.\nfor await (const testcase of client.testcases.list('246')) {\n console.log(testcase.id);\n}"
- lang: Python
source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\npage = client.testcases.list(\n testset_id=\"246\",\n)\npage = page.data[0]\nprint(page.id)"
- lang: cURL
source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID/testcases \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\""
tags:
- Testsets
post:
operationId: createTestcases
summary: Create multiple Testcases
description: Create multiple Testcases in the specified Testset.
parameters:
- in: path
name: testsetId
description: The ID of the Testset to add the Testcases to.
schema:
type: string
example: '246'
required: true
requestBody:
content:
application/json:
schema:
type: object
properties:
items:
type: array
items:
type: object
properties:
jsonData:
type: object
additionalProperties: true
description: The JSON data of the Testcase, which is validated against the Testset's schema.
required:
- jsonData
minItems: 1
maxItems: 100
description: Testcases to create (max 100).
required:
- items
examples:
Create multiple Testcases:
value:
items:
- jsonData:
question: What is the capital of France?
idealAnswer: Paris
provenance: hand_curated
- jsonData:
question: What is the largest planet in our solar system?
idealAnswer: Jupiter
provenance: synthetic
- jsonData:
question: How many planets are in our solar system?
idealAnswer: 8
provenance: user_feedback
summary: Create multiple Testcases
description: Create multiple Testcases in a single request.
responses:
'201':
description: Testcases created successfully
content:
application/json:
schema:
type: object
properties:
items:
type: array
items:
$ref: '#/components/schemas/Testcase'
required:
- items
examples:
Created Testcases response:
value:
items:
- id: '123'
testsetId: '246'
jsonData:
question: What is the capital of France?
idealAnswer: Paris
provenance: hand_curated
inputs:
question: What is the capital of France?
expected:
idealAnswer: Paris
- id: '124'
testsetId: '246'
jsonData:
question: What is the largest planet in our solar system?
idealAnswer: Jupiter
provenance: synthetic
inputs:
question: What is the largest planet in our solar system?
expected:
idealAnswer: Jupiter
- id: '125'
testsetId: '246'
jsonData:
question: How many planets are in our solar system?
idealAnswer: 8
provenance: user_feedback
inputs:
question: How many planets are in our solar system?
expected:
idealAnswer: 8
validationErrors:
- path: /data/idealAnswer
message: Expected string, received number
summary: Created Testcases
description: Example response showing successfully created Testcases.
'401':
$ref: '#/components/responses/UnauthenticatedError'
'500':
$ref: '#/components/responses/ServiceError'
x-codeSamples:
- lang: JavaScript
source: "import Scorecard from 'scorecard-ai';\n\nconst client = new Scorecard({\n apiKey: process.env['SCORECARD_API_KEY'], // This is the default and can be omitted\n});\n\nconst testcase = await client.testcases.create('246', {\n items: [\n {\n jsonData: {\n question: 'What is the capital of France?',\n idealAnswer: 'Paris',\n provenance: 'hand_curated',\n },\n },\n {\n jsonData: {\n question: 'What is the largest planet in our solar system?',\n idealAnswer: 'Jupiter',\n provenance: 'synthetic',\n },\n },\n {\n jsonData: {\n question: 'How many planets are in our solar system?',\n idealAnswer: 8,\n provenance: 'user_feedback',\n },\n },\n ],\n});\n\nconsole.log(testcase.items);"
- lang: Python
source: "import os\nfrom scorecard_ai import Scorecard\n\nclient = Scorecard(\n api_key=os.environ.get(\"SCORECARD_API_KEY\"), # This is the default and can be omitted\n)\ntestcase = client.testcases.create(\n testset_id=\"246\",\n items=[{\n \"json_data\": {\n \"question\": \"What is the capital of France?\",\n \"idealAnswer\": \"Paris\",\n \"provenance\": \"hand_curated\",\n }\n }, {\n \"json_data\": {\n \"question\": \"What is the largest planet in our solar system?\",\n \"idealAnswer\": \"Jupiter\",\n \"provenance\": \"synthetic\",\n }\n }, {\n \"json_data\": {\n \"question\": \"How many planets are in our solar system?\",\n \"idealAnswer\": 8,\n \"provenance\": \"user_feedback\",\n }\n }],\n)\nprint(testcase.items)"
- lang: cURL
source: "curl https://api2.scorecard.io/api/v2/testsets/$TESTSET_ID/testcases \\\n -H 'Content-Type: application/json' \\\n -H \"Authorization: Bearer $SCORECARD_API_KEY\" \\\n -d '{\n \"items\": [\n {\n \"jsonData\": {\n \"question\": \"bar\",\n \"idealAnswer\": \"bar\",\n \"provenance\": \"bar\"\n }\n },\n {\n \"jsonData\": {\n \"question\": \"bar\",\n \"idealAnswer\": \"bar\",\n \"provenance\": \"bar\"\n }\n },\n {\n \"jsonData\": {\n \"question\": \"bar\",\n \"idealAnswer\": \"bar\",\n \"provenance\": \"bar\"\n }\n }\n ]\n }'"
tags:
- Testsets
components:
responses:
ServiceError:
description: An internal service error indicating an issue with the Scorecard service.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiError'
examples:
Internal error:
value:
code: INTERNAL_ERROR
message: An unexpected error occurred while processing your request.
details: {}
summary: Internal error
description: Generic error when an unexpected internal issue occurs.
UnauthenticatedError:
description: Error indicating that the request is not authenticated.
content:
application/json:
schema:
$ref: '#/components/schemas/ApiError'
examples:
Authentication failure:
value:
code: UNAUTHORIZED
message: Invalid or missing authentication token
details: {}
summary: Authentication failure
description: Error returned when authentication credentials are invalid or missing.
schemas:
Testset:
type: object
properties:
id:
type: string
description: The ID of the Testset.
name:
type: string
description: The name of the Testset.
description:
type: string
description: The description of the Testset.
jsonSchema:
type: object
description: The JSON schema for each Testcase in the Testset.
additionalProperties: true
fieldMapping:
type: object
properties:
inputs:
type: array
items:
type: string
description: Fields that represent inputs to the AI system.
expected:
type: array
items:
type: string
description: Fields that represent expected outputs.
metadata:
type: array
items:
type: string
description: Fields that are not inputs or expected outputs.
required:
- inputs
- expected
- metadata
description: Maps top-level keys of the Testcase schema to their roles (input/expected output). Unmapped fields are treated as metadata.
required:
- id
- name
- description
- jsonSchema
- fieldMapping
description: 'A collection of Testcases that share the same schema.
Each Testset defines the structure of its Testcases through a JSON schema.
The `fieldMapping` object maps top-level keys of the Testcase schema to their roles (input/expected output).
Fields not mentioned in the `fieldMapping` during creation or update are treated as metadata.
## JSON Schema validation constraints supported:
- **Required fields** - Fields listed in the schema''s `required` array must be present in Testcases.
- **Type validation** - Values must match the specified type (string, number, boolean, null, integer, object, array).
- **Enum validation** - Values must be one of the options specified in the `enum` array.
- **Object property validation** - Properties of objects must conform to their defined schemas.
- **Array item validation** - Items in arrays must conform to the `items` schema.
- **Logical composition** - Values must conform to at least one schema in the `anyOf` array.
Testcases that fail validation will still be stored, but will include `validationErrors` detailing the issues.
Extra fields in the Testcase data that are not in the schema will be stored but are ignored during validation.'
Testcase:
type: object
properties:
id:
type: string
description: The ID of the Testcase.
testsetId:
type: string
description: The ID of the Testset this Testcase belongs to.
jsonData:
type: object
additionalProperties: true
description: The JSON data of the Testcase, which is validated against the Testset's schema.
inputs:
type: object
additionalProperties: true
description: Derived from data based on the Testset's fieldMapping. Contains all fields marked as inputs, including those with validation errors.
expected:
type: object
additionalProperties: true
description: Derived from data based on the Testset's fieldMapping. Contains all fields marked as expected outputs, including those with validation errors.
validationErrors:
type: array
items:
type: object
properties:
path:
type: string
description: JSON Pointer to the field with the validation error.
example: /data/question
message:
type: string
description: Human-readable error description.
example: Required field missing
required:
- path
- message
description: Validation errors found in the Testcase data. If present, the Testcase doesn't fully conform to its Testset's schema.
required:
- id
- testsetId
- jsonData
- inputs
- expected
description: 'A test case in the Scorecard system. Contains JSON data that is validated against the schema defined by its Testset.
The `inputs` and `expected` fields are derived from the `data` field based on the Testset''s `fieldMapping`, and include all mapped fields, including those with validation errors.
Testcases are stored regardless of validation results, with any validation errors included in the `validationErrors` field.'
ApiError:
type: object
properties:
code:
type: string
message:
type: string
details:
type: object
additionalProperties: true
x-stainless-any: true
required:
- code
- message
- details
description: An API error.
securitySchemes:
ApiKeyAuth:
type: http
scheme: bearer
bearerFormat: starts with ak_