Sliding-window KV caches break unless you pin the first tokens

The obvious way to stream a long conversation through a fixed-size KV cache is a sliding window: keep the most recent N tokens, drop the rest. It seems safe. Recent context is what matters, and the model has a limited window anyway.

It falls apart the moment the very first tokens get evicted. Not degrades, collapses. Perplexity jumps.

The reason is that transformers park a large share of their attention on the first few tokens regardless of what those tokens say. They are not semantically important. They are a place to put attention when a head has nothing it particularly wants to look at, and softmax forces the row to sum to one somewhere. Xiao et al. named these attention sinks, and the fix follows directly: keep the initial tokens pinned in the cache and slide the window over everything after them.

Their reported numbers are worth knowing: up to 22.2x faster than recomputing a sliding window, and stable language modeling out to four million tokens.

Two things I took from this.

The first is diagnostic. If a long-running agent gets steadily worse and the degradation correlates with session length rather than with what is being discussed, look at the cache eviction policy before you look at the prompt. "It got dumber after a while" is a systems symptom at least as often as it is a prompting one.

The second is that this is a good reminder to look at attention weights when something is strange. The sink behavior is not subtle once you plot it. It was sitting in plain sight in every attention map anyone had ever produced, and it still took a paper to say out loud that those weights are structural rather than meaningful.