LAMBADA & MMLU Benchmark — Documentation

Small language models evaluated across nine measurable stages over the OpenRouter API. This page covers the workflow, what each metric means, and the measured results.

Generated 2026-09-09 17:28 from results/v2. The full academic write-up is in extended-final-report.md.

1. Process Workflow

One benchmark run, end to end. Stages 1-9 are produced by every run; stage 10 is excluded because nothing in it is observable through a hosted inference API.

Stage 10 — hardware and distributed execution (TP, PP, DP, SP, CP, EP; VRAM; energy) — sits outside this flow. Those are configuration variables of a serving deployment, not properties of a model, and a hosted API exposes none of them. They are recorded in the run config and would only be measurable on a self-hosted vLLM path.

1.1 The two measurement passes

Each item is sent to the API twice, and the distinction explains small differences between tables.

Time-to-first-token cannot be observed from a single blocking response, so latency requires streaming. Because the two passes are separate calls, their accuracies differ slightly even at temperature 0; that residual is serving non-determinism and the web view states the delta explicitly.

2. Metric Components

Every metric the benchmark produces, grouped by stage. Observable records whether this deployment — a client of a hosted inference API — can measure it at all.

2.1 Stage 1 — Task Quality

Measure model capability. Priority P0; observable through OpenRouter: yes.

2.2 Stage 2 — Probabilistic Quality

Measure confidence and probability quality. Priority P1; observable through OpenRouter: conditional; requires provider that returns token log-probabilities.

2.3 Stage 3 — Reasoning & Consistency

Measure stability and reasoning reliability. Priority P1; observable through OpenRouter: yes; requires repeats > 1 and/or multiple seeds.

2.4 Stage 4 — Context Behavior

Measure context usage and dependency. Priority P1; observable through OpenRouter: yes; requires context ablation pass.

2.5 Stage 5 — Robustness

Measure resistance to prompt and input changes. Priority P1; observable through OpenRouter: yes; requires robustness pass.

2.6 Stage 6 — API Performance

Measure inference-service performance. Priority P0; observable through OpenRouter: yes.

2.7 Stage 7 — Token Efficiency

Measure token consumption and efficiency. Priority P0; observable through OpenRouter: yes.

2.8 Stage 8 — Economics

Measure monetary efficiency. Priority P0; observable through OpenRouter: yes.

2.9 Stage 9 — Reliability

Measure operational stability. Priority P0; observable through OpenRouter: yes.

2.10 Tokenization (LAMBADA)

Separates a tokenizer handicap from a comprehension gap: a multi-token target must be produced correctly several times over.

2.11 Stage 10 — Hardware & Distributed (excluded)

The benchmark is a client of a shared, auto-scaled third-party endpoint. Parallelism layout, GPU telemetry and power draw are chosen and held by the provider; none of them is exposed to the caller. Any figure reported here would be fabricated.

To measure it: Run the same suite against a self-hosted vLLM backend (benchmark_config.local_vllm_config), where TP/PP/DP/SP/CP/EP become settable variables and VRAM, KV cache, communication overhead and energy become directly measurable.

3. Results

Measured values per model, with a short note on where each figure comes from and why it landed where it did.

3.1 MMLU

2,850 questions per model · 3 models · 8,550 scored items. Greedy decoding (temperature 0.0).

Task quality

Serving, tokens and cost

Reliability

Composite ranking

The nominal weighting is 0.50·quality + 0.15·calibration + 0.15·robustness + 0.10·efficiency + 0.10·reliability. Calibration and robustness could not be measured in this run, so their weight is redistributed proportionally over the components that were — scoring an unmeasured component zero would penalise a model for a study that was never run. The weights above are the ones actually applied.

Why these numbers

• Ministral-8B leads at 79.3% (95% CI 77.8–80.8%). Accuracy is scored over items the API actually answered; a request refused with a 429 says nothing about the model, so it is excluded from the denominator and reported separately as an error rate.
• Llama-3.2-3B returns no parseable answer on 11.7% of items. That is an instruction-following failure, not ignorance — its score understates what it knows. Measured by parsing each response for an answer token and counting the misses separately from wrong answers, because the remedies differ.
• Cost is provider-reported per request, not estimated. Gemma-3-4B gives the cheapest correct answer at $0.000046, against $0.000083 for Llama-3.2-3B (1.8×).
• Latency is client-measured from the streamed response. Llama-3.2-3B is fastest end-to-end at 0.652 s mean; p95 is reported alongside because the tail, not the mean, is what a user experiences.
• Blank stages are absences, not zeros. Probabilistic quality, reasoning and consistency, context behaviour, robustness carry no data here: calibration needs a provider that returns token log-probabilities, and this run was routed to Cloudflare, DeepInfra, Mistral; reasoning and consistency, context behaviour, robustness each need an additional pass over the dataset, which multiplies the API spend. Each is stored as unavailable with its reason rather than filled with 0, so a gap in coverage can never be mistaken for a poor score. The History view can run them after the fact via Secondary analysis.

3.2 LAMBADA

1,000 passages per model · 3 models · 3,000 scored items. Greedy decoding (temperature 0.0).

Task quality

Context behaviour — accuracy by passage-length quartile. A flat row means length is not the constraint.

Tokenization — how much of the gap is the vocabulary rather than comprehension.

Serving, tokens and cost

Reliability

Composite ranking

The nominal weighting is 0.50·quality + 0.15·calibration + 0.15·robustness + 0.10·efficiency + 0.10·reliability. Calibration and robustness could not be measured in this run, so their weight is redistributed proportionally over the components that were — scoring an unmeasured component zero would penalise a model for a study that was never run. The weights above are the ones actually applied.

Why these numbers

• Ministral-8B leads at 39.9% (95% CI 36.9–43.0%). Accuracy is scored over items the API actually answered; a request refused with a 429 says nothing about the model, so it is excluded from the denominator and reported separately as an error rate.
• Cost is provider-reported per request, not estimated. Ministral-8B gives the cheapest correct answer at $0.000096, against $0.000107 for Gemma-3-4B (1.1×). Gemma-3-4B has the cheapest tokens but not the cheapest answers — accuracy converts token price into value.
• Latency is client-measured from the streamed response. Llama-3.2-3B is fastest end-to-end at 0.354 s mean; p95 is reported alongside because the tail, not the mean, is what a user experiences.
• Blank stages are absences, not zeros. Probabilistic quality, reasoning and consistency, robustness carry no data here: calibration needs a provider that returns token log-probabilities, and this run was routed to Cloudflare, DeepInfra, Mistral; reasoning and consistency, robustness each need an additional pass over the dataset, which multiplies the API spend. Each is stored as unavailable with its reason rather than filled with 0, so a gap in coverage can never be mistaken for a poor score. The History view can run them after the fact via Secondary analysis.
• Passage length is not the binding constraint. Sorting passages by true word count into quartiles moves accuracy by at most 8.8% (Llama-3.2-3B), and not monotonically. §4.4 tests this rather than eyeballing it: length is significant but negligible in size (V = 0.068), while target fragmentation is two to three times larger. The failures are a vocabulary handicap, not a lost-in-the-middle effect. Lengths come from the reconstructed dataset split — the per-item records store only a fixed-length preview, whose word count is not passage length (see §4.6).
• Part of the LAMBADA spread is the tokenizer. On targets needing 3+ tokens, Ministral-8B holds 34.5% while Llama-3.2-3B falls to 2.6% — a mechanical handicap, since a multi-token word must be produced correctly several times over. Note Gemma-3-4B, Llama-3.2-3B fell back to a generic BPE vocabulary (their tokenizers are gated), so their fragmentation rates are indicative rather than exact.

4. Statistical Significance and Variance

An accuracy table says Ministral scored 79.3% and Gemma 60.4%. It does not say whether that gap could be noise, nor which of the things that vary across items actually moves the result. Both are computed from the per-item records.

4.1 Method

All statistics are pure-stdlib implementations (metrics/significance.py), pinned against SciPy and statsmodels by tests/test_significance.py so the report can be regenerated on a host where SciPy cannot be installed.

4.2 Are the model differences real?

MMLU — Cochran's Q = 594.659 (df 2), p < 0.0001. The models differ; pairwise tests follow.

LAMBADA — Cochran's Q = 173.547 (df 2), p < 0.0001. The models differ; pairwise tests follow.

4.3 Do the models fail on the same items?

High overlap means the items are hard; disjoint failures mean the models have different competences and routing would pay.

4.4 Which factors move the outcome?

Significance and size are different questions. At thousands of items almost any factor reaches p < 0.05, so Cramér's V decides whether it matters.

MMLU — accuracy by subject category

MMLU — accuracy by correct-option position

LAMBADA — accuracy by passage length (true)

4.5 How much of the MMLU subject spread is real?

Under a null of identical subjects this share averages about 0.08 and rarely passes 0.3, so 0.92 is decisive: subject difficulty is a real, large effect and the per-subject table can be read. Hardest: moral_scenarios (37%), abstract_algebra (39%), high_school_physics (41%). Easiest: high_school_psychology (87%), marketing (86%), high_school_government_and_politics (85%).

4.6 What this changes

• Every pairwise gap survives correction on both benchmarks. The ranking is not an artefact of sampling — at these item counts the differences are far larger than the paired intervals.
• Tokenization matters; passage length barely does. Length reaches significance (p = 0.0031) but with V = 0.068 — negligible, and the accuracy is not even monotonic in length. Target fragmentation reaches V = 0.2058 on the worst-affected model, two to three times the length effect. The LAMBADA gap is mostly a vocabulary handicap, not a context-window one.
• Subject category is significant but small. p < 0.0001 with V = 0.0935 (negligible) across an 11.3% spread — a good illustration of why V is reported: with 8,550 pooled items, significance alone would have overstated it.
• A measurable option-position effect. Accuracy varies with where the correct answer sits: answer = C scores 68.1%, answer = D 61.1% (p < 0.0001, V = 0.0559). Small, but it is a property of the harness, not the knowledge being tested — which is why the robustness stage permutes options.
• MMLU: routing headroom of 8.4%. Some model answers 87.8% of items correctly, against 79.4% for the best single model. The failures are only partly shared — 12.2% defeat all three.
• LAMBADA: routing headroom of 12.0%. Some model answers 51.5% of items correctly, against 39.5% for the best single model. The failures are only partly shared — 48.5% defeat all three.
• One correction. Section 3's context-behaviour table previously bucketed items by the length of a stored preview string, which is truncated to a fixed 203 characters — so it ranked items by mean word length and correlated −0.20 with true passage length. It has been recomputed from the reconstructed dataset split (verified target-by-target against the stored results), and metrics/context.py now refuses to compute the metric from a preview at all. The conclusion was unchanged, but it had been reached from the wrong variable.