LLM Architecture: What Is the Brain, Really?
Tokenization, embeddings, and self-attention — Q, K, V explained with a story you will never forget. The 10-minute foundation that makes quantization, fine-tuning, and every model card readable.
What you'll learn
- Follow text through tokenizer → embeddings → attention
- Build intuition for Query, Key and Value matrices
- Know what the E4B in Gemma 4 E4B actually means
Follow your text through the pipeline
Type anything, then walk it through the four stages every prompt goes through: text → tokens → embeddings → attention.
9 tokens. Real tokenizers (BPE) learn these pieces from data — common words stay whole, rare words get split (the ## marks a continuation). Each token is just an ID number in a vocabulary.
Q, K, V — the party introduction
Self-attention has three matrices with intimidating names. The course video explains them as one moment at a party — you, asking a room full of strangers about someone named Dancy:
Query — the question you ask
“Do you know Dancy?”
You walk into a party looking for information. Your Query is what you are looking for, broadcast to everyone in the room.
Key — how each person answers
“I know Dancy!”
Every other guest holds up a Key — a label of what they can offer. Your Query is matched against every Key; strong matches score high.
Value — what they actually tell you
All the details about Dancy.
From the guests whose Keys matched, you collect their Values — the actual content — weighted by how well each one matched.
That's it: Query = what this token is looking for, Key = what each token advertises, Value = what each token actually contributes. Attention(Q, K, V) = softmax(QKᵀ/√d) · V is just this party, written in matrix form.
So what is "Gemma 4 E4B"?
The brain this course runs is Gemma 4 E4B: about 8B parameters total, but with an architecture where only ~4.5B are effective per step (that's the "E" — effective 4B), a 128K-token context window, and small enough to run in ~5GB of RAM once quantized. Every parameter is a number in one of the matrices you just explored — which raises the next question: how do you store billions of numbers cheaply? That's FP16 and quantization, the next two lessons.