Multi-Head Attention: Full Input Projection Not Slicing
Summary
I used to picture multi-head attention as cutting the embedding into pieces, one slice per head. That's wrong. Every head reads the whole input embedding, all 768 dimensions of it, and projects it down through its own learned weights. No head works with less of the input. What separates the heads is learned, not carved out of the embedding.
I had multi-head attention wrong in my head for an embarrassingly long time. With 768 embedding dimensions and 12 heads, I assumed each head got its own 64-dimensional chunk of the input: head one takes dims 1-64, head two takes dims 65-128, and so on down the line. Tidy little picture. Wrong, though.
Every head sees the whole 768-dimensional embedding, all of it, every time. Each one has its own learned Q, K, and V weight matrices, and those project the full input down to 64 dimensions. All 12 heads run attention in parallel, each in its own 64-dim space. The outputs get concatenated (12 x 64 = 768 again) and a final linear layer mixes information across heads.
Where did my slicing idea come from? Probably the phrase "different parts of the feature space," which I somehow read as "different parts of the input," as if the embedding were a pizza and every head got a slice. It means learned representations. Twelve jurors hear the same testimony, and each one latches onto a different detail. You don't hand juror number seven a transcript with eleven twelfths redacted.
Doing it this way buys you cheap parallel 64-dim math plus 12 separately trained ways of looking at the same data. My sliced-up version would have been cheaper still, and dumber.