AI Foundationsintermediate13 min · 9 steps
Attention, Self-Attention and Multi-Head Attention
How a model learns where to look: queries score keys, softmax turns scores into weights, and many heads look in different ways at once.
datarequest
ProblemStep 1 of 9
One vector can't hold a whole sentence
Early neural translators read the source sentence with a recurrent encoder and squeezed it into one fixed-size vector. The decoder then had to write the whole translation from that single summary.
Short sentences survived. Long ones lost detail — by the time the encoder reached the last word, the first ones had faded.
Watch outA fixed-size summary means a fixed amount of memory, no matter how long the input is.
Lesson 21 of 34 in AI Foundations