GGUF Quantization: 4GB → 1GB
The scale-and-round math that shrinks a model 4x with barely a quality dent: compute the scale, quantize the block to int4, dequantize on the fly. Do the arithmetic yourself on a real weight matrix.
This lesson's full video is free to watch on the homepage — watch it now and see this exact math done on the whiteboard.
What you'll learn
- Compute block scales and quantized int values by hand
- Understand Q4 / Q8 trade-offs in the GGUF format
- Shrink a checkpoint 4x and know exactly what you lost
The whole idea in three lines
scale = max(block) / 15 # 15 = biggest value 4 bits can hold q = round(weight / scale) # quantize: FP16 -> int4 (this is what gets stored) w' = q * scale # dequantize on the fly at inference time
Weights are grouped into small blocks. Each block stores one FP16 scale plus one tiny integer per weight. 16 bits per weight becomes 4 — a 4GB model becomes roughly 1GB, and the reconstruction error is bounded by half a scale step per weight.
Quantization playground
One block of eight real weights. Every number below — scale, ints, dequantized values, errors — is computed live from the formulas above. Shuffle the block, flip Q4 ↔ Q8, and watch what you lose.
One FP16 scale is stored per block; every weight in the block shares it.
| weight (FP16) | round(w / scale) | stored int4 | dequantized (q × scale) | error |
|---|---|---|---|---|
| 0.1200 | 3.000 → 3 | 3 | 0.1200 | +0.0000 |
| 0.4500 | 11.250 → 11 | 11 | 0.4400 | -0.0100 |
| 0.3000 | 7.500 → 8 | 8 | 0.3200 | +0.0200 |
| 0.6000 | 15.000 → 15 | 15 | 0.6000 | +0.0000 |
| 0.0500 | 1.250 → 1 | 1 | 0.0400 | -0.0100 |
| 0.2400 | 6.000 → 6 | 6 | 0.2400 | +0.0000 |
| 0.5100 | 12.750 → 13 | 13 | 0.5200 | +0.0100 |
| 0.3300 | 8.250 → 8 | 8 | 0.3200 | -0.0100 |
4 bits
per weight, instead of 16 — a 4x shrink
0.0200
worst error in this block
0.0075
mean absolute error
Flip between Q4 and Q8 and watch the error column: Q8 has 17x more levels to snap to, so its errors are tiny — but the file is twice the size. Choosing a quant is choosing where you sit on that trade-off.
Why this matters for agents
GGUF is the file format that packages these quantized blocks (plus the tokenizer and metadata) into a single file that llama.cpp can memory-map and run on a CPU. It is the reason Gemma 4 E4B — 8B parameters — fits in about 5GB of RAM on the free Colab T4 that powers every agent in this course. No quantization, no local agents. In the next lesson you'll meet the tool that serves these GGUF files with one command: Ollama.