Evals

MMLU Benchmark

MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 subjects from STEM and international law to nutrition and religion. It contains 15,908 multiple-choice questions (four options each), of which 1,540 are reserved for hyperparameter tuning. Per its overview, "It was one of the most commonly used benchmarks for comparing the capabilities of large language models, with over 100 million downloads as of July 2024."

GitHub

SDKs

📦
GitHubRepository
https://github.com/hendrycks/test

Other Resources

🔗
Paper
https://arxiv.org/abs/2009.03300
🔗
Dataset
https://huggingface.co/datasets/cais/mmlu

Work with this as data

Every API here is available over the APIs.io API and to AI agents over MCP.

MCP server

One button, every client — Claude, Cursor, VS Code and the rest.

https://apis.io/mcp

Tools for apis

7 MCP tools reach this
  • find_apisBrowse and filter every API in the catalog.
  • get_api_artifactsOne API's artifacts, grouped by type.
  • get_openapiThe primary OpenAPI for this API.
  • find_similar_apisAPIs that look like this one.
  • apis_io_searchSTART HERE — APIs, providers and tags for one query, each with its total.
  • resolveTurn a domain, URL or GitHub org into the provider it belongs to.
  • find_cohortsEvery scored population of providers in the catalog.
All 92 tools →

Call it yourself

curl for this page
This API 
curl "https://apis.io/api/v1/apis/mmlu-benchmark"
All apis 
curl "https://apis.io/api/v1/apis?limit=25"

Discovery needs no key. Ratings and market analysis are Pro.

Get an API key

Free tier, no form to fill in. Signing in shares your email address with us — we store it to create your key and to recognise you if you sign in with another provider. See our Privacy Policy and Terms.

GitHub Google LinkedIn

A second provider on the same verified email joins the account you already have.

API entry from apis.yml

apis.yml Raw ↑ 
name: MMLU Benchmark
description: MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice benchmark
  spanning 57 subjects from STEM and international law to nutrition and religion. It contains 15,908 multiple-choice
  questions (four options each), of which 1,540 are reserved for hyperparameter tuning. Per its overview,
  "It was one of the most commonly used benchmarks for comparing the capabilities of large language models,
  with over 100 million downloads as of July 2024."
humanURL: https://github.com/hendrycks/test
baseURL: https://github.com/hendrycks/test
tags:
- Benchmarks
- Knowledge
- Multiple Choice
- Multitask
- Reference-Based
tags_raw:
- Benchmark
- Knowledge
- Multiple Choice
- Multitask
- Reference-Based
properties:
- type: GitHubRepository
  url: https://github.com/hendrycks/test
- type: Paper
  url: https://arxiv.org/abs/2009.03300
- type: Dataset
  url: https://huggingface.co/datasets/cais/mmlu