# Shared Global KV with Layer-Specific Local History

arXiv: https://arxiv.org/abs/2609.28006 · HTML: https://mlsanity.com/paper/2609.28006/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://mlsanity.com/paper/2405.12981/) (2024)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [Attention Residuals](https://mlsanity.com/paper/2603.15031/) (2026)
- [Layer-Condensed KV Cache for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2405.10637/) (2024)
- [Block Transformer: Global-to-Local Language Modeling for Fast Inference](https://mlsanity.com/paper/2406.02657/) (2024)
- [Parallel Loop Transformer for Efficient Test-Time Computation Scaling](https://mlsanity.com/paper/2510.24824/) (2025)
- [GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression](https://mlsanity.com/paper/2407.12077/) (2024)
- [A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference](https://mlsanity.com/paper/2410.14442/) (2024)
- [Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers](https://mlsanity.com/paper/2512.03870/) (2025)
- [Inference-Friendly Models With MixAttention](https://mlsanity.com/paper/2409.15012/) (2024)
- [POP: Prefill-Only Pruning for Efficient Large Model Inference](https://mlsanity.com/paper/2602.03295/) (2026)
- [UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification](https://mlsanity.com/paper/2605.06221/) (2026)
- [You Only Index Once: Cross-Layer Sparse Attention with Shared Routing](https://mlsanity.com/paper/2606.06467/) (2026)

## Abstract (arXiv)

Decoder-only Transformer language models cache keys and values (KV) to reuse past computation during generation. Sharing KV across layers saves storage but reduces the diversity of representations available across depth. We study what local memory should retain alongside shared global KV, separating historical content from the input source used to form it. At 126M parameters and 2K context, an eight-seed study finds about 1.4% lower held-out test perplexity with local history than with a current-token local branch. Capacity, entry-count and training-compute controls support the value of historical content. In a two-seed comparison, this value persists when adjacent layers share local inputs while retaining independent projections; source sharing also shortens exact cache-construction dependencies. Against GQA and adjacent-layer KV sharing, equal bounded learning-rate searches and new-seed confirmation yield better same-source likelihood with larger caches and higher long-request latency. The ordering against adjacent-layer sharing persists after equal-token adaptation to 8K, with a short-context cost. The eight-seed external-book history effect remains uncertain, and downstream outcomes vary by task. We derive a sufficient suffix schedule that reduces upper-layer construction work while preserving the complete cache in exact arithmetic.