How we track benchmarks (and why some scores stay blank)

Correction, 20 September 2026: when this post was published it said every benchmark row carried a source_type and a last_verified_at date. That was not true of the data. Of 227 rows, 113 had no source_type at all and 87 had no source link, yet all of them displayed a verification date. We have now re-checked the whole table against the model cards it cites; the rest of this post describes the rules, and the section below describes what the check found.

What the September 2026 re-check found

Every row was screened against the Hugging Face model card it cited, and the 40 rows whose number actually appeared in a readable card were then adjudicated line by line. 29 rows are confirmed: the card reports that number, for that benchmark, for that model. 11 rows were wrong and have been deleted — they had picked up a different benchmark's value, or a competitor's column from a comparison table. Three more held the wrong figure and have been corrected against the card. The remaining 187 rows are now labelled 'unconfirmed' and carry no verification date, because we could not confirm them at a source we can reach.

Two things made this worse than it looked. Twenty of the model cards we cite are gated, so a reader following our own source link hits an access wall. And deepseek-ai/DeepSeek-R1-Distill-Llama-70B ships a byte-identical copy of the DeepSeek-R1 card, so R1's headline numbers sit right there in the distill's card waiting to be misread as the distill's own — four of that model's five rows failed for exactly that reason.

Rule one: only what the lab publishes

For every benchmark score we display, someone with credibility (the lab, the paper authors, the provider) has published the number. If we can't find an official source, the cell stays NULL — not zero, not a guess. Where we hold a number but can't point at the source, we now say so on the row rather than implying a check we didn't do.

This means the site under-reports. Llama 3.2 1B's HumanEval score is real; the fact that some benchmarks appear blank for smaller models is because those labs didn't run those evals. We don't fill the gaps.

Rule two: dates every time

Benchmarks age. HumanEval was frontier in 2022 and is now saturating for frontier models. MATH scores that looked impressive in 2024 look unremarkable in 2026. A row carries a last_verified_at only when someone actually confirmed the number at its source on that date — a date is a record of a check, not a decoration. Rows we haven't been able to confirm show no date, which is the honest answer.

Rule three: no aggregate scores across families

Different labs use different eval harnesses, different prompt formats, different few-shot counts. Comparing MMLU scores across families is at best directional. We show the numbers because they're useful signal; we don't average them or produce a composite 'quality' rank.

Rule four: the leaderboard reflects reality, not our opinions

SWE-bench Verified is more predictive of real coding-agent quality than HumanEval. IFEval matters more for chat workloads than MMLU. ArenaHard correlates with human preference. That's why we surface those alongside the classical academic set. When a model doesn't publish a score, its row is absent — silence, not zero.

For the full methodology see /methodology. For the raw benchmark data see /leaderboards. For corrections, open a PR at github.com/bryanflowers/opensourceaimodels.