Skip to content
anythingai.space
Training & Adaptationintermediate13 min · 9 steps

RLHF and DPO

How preference data steers a model: the classic RLHF pipeline, why DPO replaced it for most teams, online alignment, and where rubric and reasoning rewards go wrong.

PromptPretrained modelknowledgeable, uncontrolledHarmful contentsafetyIgnores the askinstruction followingHallucinatesfactuality
requesterrordataresponsesignalheavy worktoken
1 / 9
ProblemStep 1 of 9

Pretrained models know a lot and obey nothing

A pretrained model is knowledgeable but uncontrolled. Ask it something and it may produce harmful content, ignore the instruction, or hallucinate freely.

Alignment steers behaviour toward human intent, using a reward signal and policy updates.

What changed
  • +Three ways an unaligned model fails

Lesson 4 of 8 in Training & Adaptation