MMLU Benchmark - Knowledge & Reasoning

Multiple-choice questions across the 57 MMLU subjects, fetched live from the free Hugging Face datasets-server API. Each model must reason step by step before answering; the reasoning is parsed, scored for consistency, and shown per question below.

Subjects
Decoding parameters

MMLU uses greedy decoding by default and a larger token budget than LAMBADA so the model has room to write its reasoning before committing to a letter. Hover a preset to see the trade-off it makes.

Randomness. 0 = always take the most likely token (best for benchmarking).
Sample only from the smallest set of tokens whose probabilities sum to p.
Consider only the k most likely tokens. 0 disables the limit.
Drop tokens below this fraction of the top token's probability.
Output budget. Too low truncates reasoning before the answer letter.
Penalises tokens by how often they have already appeared.
Penalises tokens that have appeared at all, regardless of count.
Divides the logit of already-seen tokens. 1.0 = off.
Fixed seed makes a run repeatable where the provider honours it. Blank = random.
Extended analysis passes 1x calls
Single-token scoring to recover ECE, Brier, NLL and target rank. Only yields numbers for providers that return log-probabilities.
Prompt variation, option reordering and typo noise, scored as accuracy deltas.
Re-runs with the passage cut down, to measure real context utilisation.
Repeat each item N times to measure answer stability and self-consistency.
Model ranking

Quick score = 0.70 × accuracy + 0.15 × reasoning consistency + 0.15 × relative speed. Per-dimension ranks are shown in parentheses (1 = best).
This is the run page's own summary and is not the composite shown in History, which is weighted over the nine-stage taxonomy and renormalised over whichever stages were measured. The two are different numbers.

# Model Accuracy (%) STEM Human. Social Other Reasoning consist. (%) Avg words Avg time (s) Quick score Errors
Accuracy by model
Accuracy by category
Questions, answers & model reasoning

Every question the selected model saw, its pick vs the correct answer, and the parsed reasoning behind it.