Generative AIbeginner12 min · 9 steps
Language Model Families
How text generation works — next-token prediction — and the families built on it: encoder-only BERT, encoder-decoder T5, the decoder-only GPT line through ChatGPT, and open LLaMA.
signalresponserequestdataheavy work
ProblemStep 1 of 9
Rules and n-grams only see a few words back
Older text generators were hand-written rules or statistical n-gram models that pick the next word from the last two or three. They have no idea what the sentence, let alone the document, is about.
Ask one to continue “The bank raised its rates, so the river…” and it can't tell which bank it's in. The meaning lives further back than its window reaches.
Watch outN-gram models are now considered outdated: they can't use long-range context.
Lesson 2 of 8 in Generative AI