System design
LLM system design interview questions: KV cache, attention, and serving tradeoffs
KV cache, attention variants, and inference cost are where LLM system design interviews go deep. Here is the chain of reasoning interviewers follow.
KV cache avoids recomputation — but that is only the first layer
Most candidates can explain that a KV cache lets a model reuse previously computed keys and values instead of recomputing them at every generation step. That is the correct starting answer, and it is also where most self-study stops.
A real interviewer’s next question is usually: what happens to that cache as context length grows? The honest answer is that KV cache memory grows linearly with sequence length and batch size, and at long context lengths it can dominate GPU memory — often becoming the actual bottleneck for serving, not compute.
Why that leads straight into attention variants
Once memory growth is on the table, the natural follow-up is: how do you reduce it? This is where multi-head attention, multi-query attention (MQA), and grouped-query attention (GQA) come in — each trades off cache size against model quality differently. MQA shares a single key/value head across all query heads, cutting cache size dramatically at some quality cost; GQA groups query heads to share fewer key/value heads, sitting between MHA and MQA.
Being able to explain why a serving team would choose GQA over MQA for a production LLM — not just what the terms mean — is usually the actual signal an interviewer is listening for.
The pattern behind this whole topic
Notice the shape: the interviewer rarely stays at the first correct answer. They follow it into what it implies for memory, then into what you would do about it. Studying the definitions is necessary but not sufficient — the gap is usually in the second or third link of that chain, not the first.
This is the exact chain Assessmentr’s diagnostic is built to probe: starting from your explanation of attention, following into inference cost, and only then to deployment tradeoffs — with a ranked gap map showing where the reasoning chain actually broke down.
Find your own gap map.
Free during the beta — one voice diagnostic, one ranked gap, one next action.