AI Foundationsbeginner14 min · 11 steps
Tokens and Embeddings
How text becomes numbers: split it into tokens, then map every token, word or sentence to a vector whose position encodes meaning.
errordatarequest
ProblemStep 1 of 11
A neural network can't read a string
Every layer of a network multiplies and adds numbers. Hand it the string “unhappiness is rare” and there is nothing to multiply.
Two jobs stand in between: cut the text into units the model has a fixed list of, then turn each unit into a vector of numbers that carries its meaning.
InsightEvery NLP system — from a spam filter to an LLM — starts with these two steps: tokenize, then embed.
← Previous lessonConvolution & PoolingNext lesson →Attention, Self-Attention and Multi-Head Attention
Lesson 20 of 34 in AI Foundations