Skip to content
anythingai.space
Evaluation & Observabilityintermediate12 min · 8 steps

Benchmarks and Leaderboards

Read a benchmark score correctly: what it measures, whether it's saturated or contaminated, and which harness produced it.

“88% on SWE-bench”WHAT THE NUMBER DOESN'T SAYversion?harness / scaffold?effort level?binary or partial?who ran it?fallback model?cost per task?
heavy workresponsedata
1 / 8
ProblemStep 1 of 8

“88% on SWE-bench” tells you almost nothing

A headline score hides how it was produced. The same weights can score very differently depending on the agent scaffold, tool access, effort level and output-token cap, and since mid-2026 also the benchmark version, the metric, who ran it and whether a fallback model handled some tasks.

Vendor self-reports also run higher than independent leaderboards, because labs report the best configuration they found.

Watch outThe 2026 web is full of fabricated leaderboard pages. If a frontier score names a model you can't find in a primary source, assume it's SEO spam.

Lesson 3 of 5 in Evaluation & Observability