03·Benchmarks & methodology
What the numbers actually mean
Benchmark names tell you nothing. Each card below says what the test really measures, what a score gap means in practice, and which of your tasks it predicts — so "GPQA 85" stops being a vibe.
MMLU-Pro
knowledgeweight 10%Multiple-choice questions across 14 broad subjects — law, medicine, business, engineering. It measures how well-read and generally competent a model is, not how deeply it can reason.
What a gap means · A 3-point gap is barely noticeable in everyday chat. Below ~75, expect visible factual slips in specialist domains.
GPQA Diamond
reasoningweight 15%PhD-level science questions written to be "Google-proof" — human experts with web access score around 65%. This is the cleanest test of genuine reasoning depth rather than memorization.
What a gap means · 5 points is roughly the gap between "usually right, hand-wavy why" and "right, with a defensible chain of logic".
Humanity's Last Exam
reasoningweight 10%2,500 frontier questions across 100+ fields, written by experts specifically to stump AI. Scores look low by design — even the best models fail most questions.
What a gap means · Every point here is hard-won. Treat 30+ as elite research-grade capability. Largely irrelevant for everyday tasks.
AIME 2025
mathweight 15%The American Invitational Mathematics Examination — 15 competition problems requiring long, exact multi-step derivations. Near-perfect scores mean real math, not pattern-matching.
What a gap means · Below ~70, don't trust unassisted quantitative derivations — check the algebra yourself. This is also the benchmark that collapses hardest when you lower reasoning effort.
SWE-bench Verified
codingweight 20%The model is dropped into a real GitHub repository and asked to fix a real reported issue; success is judged by the repo's own test suite. The single best proxy for "can it do my coding tickets end-to-end".
What a gap means · 5 points ≈ one more ticket resolved out of every twenty, with no human help.
LiveCodeBench
codingweight 10%Fresh competitive-programming problems published after each model's training cutoff — algorithmic coding with zero chance of contamination.
What a gap means · Predicts self-contained coding: writing functions, small scripts, tricky algorithms. Less predictive of large-codebase work than SWE-bench.
τ²-bench
agenticweight 10%The model plays a customer-service agent that must use tools, follow company policy, and talk to a simulated human across many turns. Measures staying reliable and on-task in long workflows.
What a gap means · Below ~70, agents drift off policy in long tool-use chains — fine for demos, risky for production automation.
IFEval
instructionweight 10%Does the model follow explicit, checkable instructions exactly — word counts, formats, forbidden words? Boring, but it decides whether outputs drop into your pipeline unedited.
What a gap means · 3 points ≈ a few more malformed outputs per hundred in a production pipeline.
A·Methodology
- Intelligence Index— the weighted composite above (weights shown per card, summing to 1). Computed separately for every (model × effort × host) configuration; there is no single "the score" for a model.
- Effort sweet spot — per task category, the cheapest effort tier retaining ≥96%of the model's own peak on that category's benchmarks.
- Drift — each tracked configuration is re-evaluated weekly on a fixed probe suite; the series you see is the index over time, with serving-change events annotated when we can attribute them.
- Task mapping — Everyday writing & Q&A, Code assist, Coding agents, Math & quantitative, Hard research problems, Agents & tool use each map to 2 benchmarks (listed on every model page).
B·Data status — read this
This is a v1 preview. Every score, price, speed and drift event on the site is hand-authored illustrative data, built to demonstrate the product: per-effort scoring, host-variant deltas, and drift tracking. None of it is a measured result, and none of it should be cited.
The continuous eval pipeline replaces this dataset in place — same schema, same pages, real weekly measurements. Until then, the honest label stays on every page: preview data.
Questions about the method? The gateway docs describe how effort and variants are addressed at the API level.