Training & Adaptationadvanced14 min · 9 steps
Quantization Deep Dive
Fewer bits per weight: how BF16 becomes FP8, 4-bit and below, which methods keep quality, why kernels decide speed, and why the KV cache matters as much as the weights.
8B weights16GB
quality loss0%
metrics are illustrative
1 / 9ProblemStep 1 of 9
Every weight costs 16 bits
Models are traditionally stored in BF16: 16 bits per weight (1 sign, 8 exponent, 7 mantissa). An 8B-parameter model is therefore 16 GB of weights before any KV cache or activations.
Quantization stores weights with fewer bits to save memory and speed up inference. It's the main tool for single-GPU and consumer deployment, and in 2026 also the datacenter default: FP4 weights with an FP8 KV cache is the headline configuration.
What changed
- +One BF16 weight
- +8B model weight size
Lesson 7 of 8 in Training & Adaptation