Skip to content
anythingai.space
AI Foundationsintermediate14 min · 10 steps

Gradient Descent & Optimizers

Walk downhill on the loss: how gradient descent, SGD, learning-rate choice, schedules, adaptive rates and Adam turn a gradient into better weights.

loss0.88
LOSS L(W) FOR ONE WEIGHT Wstart (random init)minimum
dataresponse
metrics are illustrative
1 / 10
ProblemStep 1 of 10

Training is a search for the lowest loss

A model's loss measures how wrong it is on the training data — mean squared error, cross-entropy. Every weight setting w gives a loss value, so the loss is a landscape over weight space.

Here the model has a single weight and the landscape is a simple bowl. Real networks have millions or billions of weights, so we cannot try every setting — we need a way to know which way is downhill from where we stand.

Lesson 17 of 34 in AI Foundations