AimRank Annotator Qualification API
The Annotator Qualification API from AimRank — 5 operation(s) for annotator qualification.
Work with this as data
Every API here is available over the APIs.io API and to AI agents over MCP.
MCP server
One button, every client — Claude, Cursor, VS Code and the rest.
https://apis.io/mcp
Tools for apis
7 MCP tools reach this
find_apisBrowse and filter every API in the catalog.get_api_artifactsOne API's artifacts, grouped by type.get_openapiThe primary OpenAPI for this API.find_similar_apisAPIs that look like this one.apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.resolveTurn a domain, URL or GitHub org into the provider it belongs to.find_cohortsEvery scored population of providers in the catalog.
Call it yourself
curl for this page
This API
curl "https://apis.io/api/v1/apis/aimrank-io:aimrank-io-annotator-qualification-api"
All apis
curl "https://apis.io/api/v1/apis?limit=25"Discovery needs no key. Ratings and market analysis are Pro.
Get an API key
Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.
A second provider on the same verified email joins the account you already have.
OpenAPI Specification
openapi: 3.2.0
info:
title: AimRank Annotator Qualification API
description: Ranking-as-a-Service API — Turn binary comparisons into calibrated rankings with Glicko-2, Bradley-Terry, TrueSkill, and cross-domain transfer learning. Includes webhooks, API keys, RLHF exports, and embeddable widgets.
version: 4.12.0
tags:
- name: Annotator Qualification
paths:
/api/v1/quality/rankings/{ranking_id}/qualification-items:
post:
tags:
- Annotator Qualification
summary: Create Qualification Item
description: Create a gold-standard qualification item. Owner-only.
operationId: create_qualification_item_api_v1_quality_rankings__ranking_id__qualification_items_post
parameters:
- name: ranking_id
in: path
required: true
schema:
type: string
title: Ranking Id
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/QualificationItemCreate'
responses:
'201':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/QualificationItemResponse'
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
get:
tags:
- Annotator Qualification
summary: List Qualification Items
description: List all qualification items for a ranking. Owner-only.
operationId: list_qualification_items_api_v1_quality_rankings__ranking_id__qualification_items_get
parameters:
- name: ranking_id
in: path
required: true
schema:
type: string
title: Ranking Id
responses:
'200':
description: Successful Response
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/QualificationItemResponse'
title: Response List Qualification Items Api V1 Quality Rankings Ranking Id Qualification Items Get
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
/api/v1/quality/rankings/{ranking_id}/qualification-items/{item_id}:
delete:
tags:
- Annotator Qualification
summary: Delete Qualification Item
description: 'Soft-delete a qualification item (sets is_active=False). Owner-only.
Soft delete preserves the audit trail of past attempts; pass status
recomputes against only `is_active=True` items.'
operationId: delete_qualification_item_api_v1_quality_rankings__ranking_id__qualification_items__item_id__delete
parameters:
- name: ranking_id
in: path
required: true
schema:
type: string
title: Ranking Id
- name: item_id
in: path
required: true
schema:
type: string
title: Item Id
responses:
'204':
description: Successful Response
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
/api/v1/quality/rankings/{ranking_id}/qualification/next:
get:
tags:
- Annotator Qualification
summary: Get Next Qualification Item
description: 'Get the next unanswered qualification item for the current voter.
Returns the item without `expected_winner_id` so the voter can''t game
the test. When all active items have been attempted, returns
`{"complete": true, ...status}`.'
operationId: get_next_qualification_item_api_v1_quality_rankings__ranking_id__qualification_next_get
parameters:
- name: ranking_id
in: path
required: true
schema:
type: string
title: Ranking Id
responses:
'200':
description: Successful Response
content:
application/json:
schema: {}
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
/api/v1/quality/rankings/{ranking_id}/qualification/attempts:
post:
tags:
- Annotator Qualification
summary: Submit Qualification Attempt
description: 'Record a voter''s answer to a qualification item.
Voters never see the correct answer in the API response — only
pass/fail status (so they can''t iterate by trial-and-error to memorise
the test bank).'
operationId: submit_qualification_attempt_api_v1_quality_rankings__ranking_id__qualification_attempts_post
parameters:
- name: ranking_id
in: path
required: true
schema:
type: string
title: Ranking Id
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/QualificationAttemptCreate'
responses:
'201':
description: Successful Response
content:
application/json:
schema: {}
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
/api/v1/quality/rankings/{ranking_id}/qualification/status:
get:
tags:
- Annotator Qualification
summary: Get Qualification Status
description: Get the current voter's qualification status for this ranking.
operationId: get_qualification_status_api_v1_quality_rankings__ranking_id__qualification_status_get
parameters:
- name: ranking_id
in: path
required: true
schema:
type: string
title: Ranking Id
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/QualificationStatusResponse'
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
components:
schemas:
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
QualificationItemCreate:
properties:
entity_a_id:
type: string
title: Entity A Id
description: One of the entities being compared
entity_b_id:
type: string
title: Entity B Id
description: The other entity
expected_winner_id:
type: string
title: Expected Winner Id
description: The canonically-correct winner (must be entity_a_id or entity_b_id)
rationale:
anyOf:
- type: string
maxLength: 1000
- type: 'null'
title: Rationale
type: object
required:
- entity_a_id
- entity_b_id
- expected_winner_id
title: QualificationItemCreate
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
QualificationItemResponse:
properties:
id:
type: string
title: Id
ranking_id:
type: string
title: Ranking Id
entity_a_id:
type: string
title: Entity A Id
entity_b_id:
type: string
title: Entity B Id
expected_winner_id:
anyOf:
- type: string
- type: 'null'
title: Expected Winner Id
rationale:
anyOf:
- type: string
- type: 'null'
title: Rationale
is_active:
type: boolean
title: Is Active
type: object
required:
- id
- ranking_id
- entity_a_id
- entity_b_id
- is_active
title: QualificationItemResponse
QualificationAttemptCreate:
properties:
qualification_item_id:
type: string
title: Qualification Item Id
given_winner_id:
type: string
title: Given Winner Id
type: object
required:
- qualification_item_id
- given_winner_id
title: QualificationAttemptCreate
QualificationStatusResponse:
properties:
ranking_id:
type: string
title: Ranking Id
voter_id:
type: string
title: Voter Id
qualification_required:
type: boolean
title: Qualification Required
pass_threshold:
type: number
title: Pass Threshold
n_items:
type: integer
title: N Items
n_attempted:
type: integer
title: N Attempted
n_correct:
type: integer
title: N Correct
accuracy:
type: number
title: Accuracy
has_passed:
type: boolean
title: Has Passed
type: object
required:
- ranking_id
- voter_id
- qualification_required
- pass_threshold
- n_items
- n_attempted
- n_correct
- accuracy
- has_passed
title: QualificationStatusResponse
securitySchemes:
APIKeyHeader:
type: apiKey
in: header
name: X-API-Key