Retrieval & RAGintermediate12 min · 9 steps
Data Engineering for AI
One pipeline feeds both RAG and fine-tuning: detect, parse, clean, dedup, govern and filter — then keep it fresh and auditable.
errordata
ProblemStep 1 of 9
Real input is a heterogeneous pile
PDFs, scans, office files, HTML, email, images. Trust a file's extension and you send it to the wrong parser; push a scanned PDF down the text path and you get empty or partial text — silently.
Most of the work in a serious AI system happens upstream of the model. The widely quoted “80% of time on data prep” is folklore-grade, but the direction is right.
Lesson 15 of 37 in Retrieval & RAG