Training & Adaptationintermediate12 min · 9 steps
Fine-Tuning Strategies
When to fine-tune, which kind (SFT, continued pretraining, PEFT), how to set it up, and where fine-tuning can still run in 2026.
requesterrordataresponseheavy work
ProblemStep 1 of 9
Fine-tuning teaches form, not facts
Before training anything, ask: can prompting or RAG solve this? Fine-tuning is now less about teaching facts and more about teaching format and behaviour.
LLMs are bad at memorising facts from fine-tuning, and RAG is far easier to update. But a reliable JSON schema, a consistent persona, or dropping a long few-shot prompt to cut latency are all good fine-tuning jobs. Specialised vocabulary calls for continued pretraining.
| Requirement | Better solution |
|---|---|
| New facts / knowledge | RAG |
| Specific output format | Fine-tuning |
| Tone / persona | Fine-tuning |
| Latency (drop few-shot prompt) | Fine-tuning |
| Private domain language | Continued pretraining |
What changed
- +Four common needs
- +The right tool for each
Lesson 2 of 8 in Training & Adaptation