models.fyi · continuous model metrology
The benchmark you read is not the model you run.
Leaderboards quote one number — max effort, official host, launch week. We measure every model at every reasoning-effort level, on every host that serves it, every week — then give you one API to call exactly the configuration you chose.
11 models · 24 serving configs · 8 benchmarks × 5 effort tiers · weekly re-runs
Intelligence vs cost · at medium effort — the tier people actually run
Cost per typical task, log scale (1K in + 600 out + reasoning tokens)
00·Why this exists
Three dimensions every other leaderboard flattens away
01 · Effort levels
minimal
low
medium
high
xhigh
Same model, five different machines
GPT-5.5 spans 62.4–82.3index points depending on the effort dial. We score every tier and compute the cheapest one that's still safe — per task type.
02 · Host variants
Same weights, different model
Open models are served at different quantizations by different hosts. GLM-5.2 is not one score — it's one score per serving. We measure each id you can actually call.
03 · Drift
"It got dumber" — with evidence
Endpoints change under you: silent checkpoints, quantization swaps, stack regressions. Weekly re-runs turn 降智 folklore into dated, attributable index drops.
04·The gateway
Analysis you can deploy
Every configuration on this site is a callable id on an OpenAI-compatible endpoint. Compare, pick the sweet spot, ship it — one key, unified billing, usage metered per model, host and effort tier.
- → drop-in: change base_url, keep your SDK
- → host pinning: model@host addresses one serving
- → effort routing: reasoning_effort is first-class
- → streaming SSE, hashed keys, per-tier metering
curl https://models.fyi/api/v1/chat/completions \
-H "Authorization: Bearer $MFYI_API_KEY" \
-d '{
"model": "glm-5.2@fireworks", ← the exact serving you compared
"reasoning_effort": "medium", ← the tier the sweet spot recommends
"messages": [{"role":"user","content":"..."}]
}'