Methodology

Scores are taken from each model's official Hugging Face card, the accompanying paper, or the release blog post — whichever the lab publishes as authoritative. If multiple sources disagree, we prefer the most recent. When a model lacks an official score for a benchmark we track (MMLU, HumanEval, MATH), we leave the field blank rather than guess.

Why this matters: labs use slightly different evaluation harnesses, prompt formats, and few-shot counts. Cross-family comparisons can over-state real-world differences. Treat the leaderboards as guidance, not gospel. For workloads where benchmark accuracy materially affects your decision, re-run the eval on your own infrastructure.

Each benchmark row stores a source_type. A row marked official_card has been read against the model card it cites — that exact number, for that benchmark, for that model — and carries the date of that check plus a link. A row marked unconfirmed is a figure we hold but have not been able to confirm at a source we can reach, and it carries no date. We would rather show you which is which than imply a check we didn't do.

Where a card reports a named variant of a benchmark we track — MATH-500 rather than MATH, GPQA-Diamond rather than GPQA, HumanEval-Mul rather than HumanEval — we keep the row in the broader bucket so leaderboards stay comparable, and show the exact label the source used underneath it. Renaming the bucket instead would split every leaderboard into one-row categories.

The headline VRAM figures on each model page are weight-storage estimates: fp16 ≈ params × 2 GB and Q4 ≈ params × 0.6 GB. These are loading-only floors — real inference also needs memory for the KV cache, activations, and CUDA / runtime overhead.

For long-context workloads, KV cache often dominates. Plan for 1.5–2× the listed VRAM at full context length. The "Recommended hardware" field on each model page accounts for typical usage and is hand-curated rather than computed.

Prices are listed in USD per million tokens for each provider's hosted endpoint, at the time the row was last verified. Providers update pricing frequently — always confirm on the provider's own pricing page before integrating. Every pricing row carries a last_verified_at date.

We attempt to list at least one provider per model that supports hosted inference. Models without listed pricing are typically self-host-only on this site — either because no major provider hosts them, or because we haven't verified pricing yet.

A price only appears on this site while the provider still publishes one. Open-weight models get retired from hosted catalogues often, so every pricing row carries a status. When a provider shuts a model down, deprecates it, or moves it behind an enterprise contract, we clear the price and keep the row — the model page then shows what happened and when, with a link to the provider's own announcement, under No longer listed. We deliberately do not republish the old rate: it was verified against a page that no longer exists, so we can't stand behind it. Delisted rows are excluded from the cheapest-hosted strip, the cost calculator, comparisons and our structured data.

The "Cheapest" label on the model detail page is computed from input price; ties broken on output. It's a starting point, not a recommendation — latency, region, and rate-limit availability matter as much as raw $/MT for production workloads.