Evaluation & Observabilityintermediate13 min · 9 steps
LLM Evaluation
Score open-ended outputs on several dimensions with code checks, LLM judges and humans, and know where each one lies to you.
requestresponsedataerrorsignal
ProblemStep 1 of 9
There's no single right answer to compare against
Classic ML has one label per input and a clean metric like accuracy or F1. An LLM writes open-ended text, and there are many valid ways to answer the same question.
Worse, quality has several independent axes. A response can be correct but poorly written, fluent but incomplete, or complete but off-topic. One score hides which axis failed.
| Traditional ML | LLM systems |
|---|---|
| Single correct answer | Many valid responses |
| Objective metrics | Subjective quality |
| Easy to automate | Needs judgment |
| Static test sets | Diverse scenarios |
InsightMeasure each dimension on its own. “Is it good?” is several questions.
Lesson 1 of 5 in Evaluation & Observability