How to Read AI Benchmarks and Leaderboards Without Being Fooled

Every new model claims to be the best. Here's how to interpret the numbers.

• Automated tests: fixed questions with known answers (maths, coding, reasoning, knowledge).
• Human preference arenas: people compare two outputs blindly, producing an Elo-style score.
• Task-based evaluations: real jobs such as fixing software issues or operating a computer.
• Vendor-run evaluations: tests the provider designed and ran.

• Contamination: the model may have seen the test questions during training.
• Saturation: when everyone scores 95%+, the test stops telling models apart.
• Narrow scope: short English clips or exam-style questions don't reflect your work.
• Gaming: models can be tuned to specific tests.
• Different set-ups: prompts, tools and settings change results.
• Snapshot timing: rankings change weekly, and different sites show different scales.
• Preference isn't accuracy: people may prefer confident, polished answers over correct ones.

1. Check who runs it and how.
2. Compare scores within one leaderboard, not across different ones.
3. Look at confidence intervals or vote counts. Small gaps may be noise.
4. See whether results are independent or vendor-reported.
5. Look at price and speed, not just score.
6. Match the benchmark to your task.

1. Collect 20 to 50 real examples from your work.
2. Run them through two or three candidate models.
3. Score blind against clear criteria: accuracy, tone, format, cost and speed.
4. Re-test when models update.

Benchmarks are a starting point, not a verdict. The best model is the one that performs reliably on your tasks at a price you can live with.