mlsanityData to 14 Oct 2026

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

arXiv 2607.23054 ↗First posted 25 Jul 2026cs.LG cs.AI cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference. Despite its adoption in massive production models, no prior work has studied what information this bottleneck preserves or discards, nor how it reshapes internal transformer circuits. We present the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer (pretrained on a web/code/math mixture, fine-tuned on TinyStories) and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis. Our key findings are: (1) the cKV bottleneck learns a pure content representation, preserving entity identity (98% retention) while discarding positional information, validating MLA's separation of content from position via RoPE; (2) induction heads co-locate at a single layer (Layer 12), unlike their distributed formation in standard MHA; (3) a single "semantic hub" layer (Layer 15) simultaneously exhibits the highest SVD effective rank and strongest disruption-attribution score; and (4) the bottleneck is globally over-provisioned, using only 46% of its capacity on average. These findings suggest MLA does not merely compress attention passively, but reshapes how the model organizes content, position, and circuit structure. We view this as an initial data point and detail scope limitations in Section 5.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 25 references that are in the map. First posted on arXiv 25 Jul 2026.

Builds on

17 of 25 references.

References · 17

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2022Toy Models of Superpositioncited by 971
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
5 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  2. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  3. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  4. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  5. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Latent Multi-Head Attention for Small Language Models
  2. 2025EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
  3. 2026Latent-Condensed Transformer for Efficient Long Context Modeling
  4. 2025Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention
  5. 2026Linear Predictability of Attention Heads in Large Language Models
  6. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
  7. 2025GTA: Grouped-head latenT Attention
  8. 2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models