Skip to content
anythingai.space
AI Foundationsbeginner14 min · 11 steps

Tokens and Embeddings

How text becomes numbers: split it into tokens, then map every token, word or sentence to a vector whose position encodes meaning.

a string?u · n · h · a · p · p · i · n · e · s · s → ???Raw text“unhappiness is rare”Neural networkexpects vectors
errordatarequest
1 / 11
ProblemStep 1 of 11

A neural network can't read a string

Every layer of a network multiplies and adds numbers. Hand it the string “unhappiness is rare” and there is nothing to multiply.

Two jobs stand in between: cut the text into units the model has a fixed list of, then turn each unit into a vector of numbers that carries its meaning.

InsightEvery NLP system — from a spam filter to an LLM — starts with these two steps: tokenize, then embed.

Lesson 20 of 34 in AI Foundations