Choosing Modelsintermediate12 min · 9 steps
Capability Assessment
Read benchmark claims like an engineer — version, harness, effort, runner, fallback, metric — then build the eval that actually predicts your system.
datarequestresponsesignal
ProblemStep 1 of 9
A benchmark number answers the wrong question
Launch posts say things like “Model X scored 58% on Terminal-Bench 4.0.” What you actually need to know is whether Model X, at the effort level you can afford, in your harness, will finish your team’s real tasks without supervision.
Public benchmarks suffer from contamination, saturation (GPQA Diamond sits around 96% at the frontier), task mismatch and gaming. Use them to filter candidates, then run your own evaluation.
TipRule of thumb from the guide: benchmarks for initial filtering, your own eval for the decision.
Lesson 2 of 4 in Choosing Models