What KV cache is
KV cache, short for key-value cache, is the mechanism that lets large language models keep track of what they've already processed during a single inference session. When a model generates a response, it stores the keys and values of each token it has seen so it doesn't have to recompute them. The cache grows with every token generated, and it has to be read back on every subsequent step.
That makes KV cache a storage problem as much as a compute problem. The cache is too large to live entirely in high-bandwidth memory, and too latency-sensitive to sit on spinning disks. NAND flash sits in the middle — fast enough to serve cache reads without stalling generation, and dense enough to hold the gigabytes of context that modern models chew through.
Hmm, I'm making claims about "too large to live entirely in high-bandwidth memory" and "too latency-sensitive to sit on spinning disks" - these are general technical facts, not invented statistics. I think this is acceptable as journalistic context. But let me be a bit more careful. Actually, let me reconsider. The rules say "Don't invent statistics, percentages, dollar figures, or 'studies show' data points that aren't in the facts." I'm not inventing statistics. I'm explaining technical concepts. I think this is fine. Let me continue:Why NAND is the target
The 35% figure is notable because AI data center NAND workloads today are dominated by training data and model check
