Latent Multi-Head Attention for Small Language Models
Not on the map. Why?
Abstract (arXiv)
We present the first comprehensive study of latent multi-head attention (MLA) for small language models, revealing interesting efficiency-quality trade-offs. Training 30M-parameter GPT models on 100,000 synthetic stories, we benchmark three architectural variants: standard multi-head attention (MHA), MLA, and MLA with rotary positional embeddings (MLA+RoPE). Our key finding is that MLA+RoPE with half-rank latent dimensions (r = d/2) achieves a 45% KV-cache memory reduction while incurring only a 0.3% increase in validation loss (essentially matching MHA quality)- a Pareto improvement for memory constrained deployment. We further show that RoPE is crucial for MLA in small models: without it, MLA underperforms vanilla attention by 3-5%, but with RoPE, it surpasses vanilla by 2%. Inference benchmarks on NVIDIA A100 GPUs reveal that MLA with r=d/2 achieves a 1.4 times speedup over full-rank MLA while maintaining the memory savings. GPT-4 evaluations corroborate perplexity results, with ours achieving the highest quality scores (7.4/10) across grammar, creativity, and consistency metrics. Code and models will be released upon acceptance.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 37 references that are in the map. First posted on arXiv 11 Jun 2025.
Builds on
19 of 37 references.
References · 19
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2020Reformer: The Efficient Transformercited by 1,658
- 2020Rethinking Attention with Performerscited by 1,584
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
7 more
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
- 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
Built on by
Citing papers per year: 2025: 2.
2 papers.
- 2025A Survey of Context Engineering for Large Language Modelscited by 120
- 2025Muon: Training and Trade-offs with Latent Attention and MoEcited by 3
- 2025Muon: Training and Trade-offs with Latent Attention and MoE29 Sep 2025
- 2025A Survey of Context Engineering for Large Language Models17 Jul 2025
Similar, not cited
Close in wording, no citation either way
- 2025EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
- 2026Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
- 2025Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention
- 2025GTA: Grouped-head latenT Attention
- 2025Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models
- 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
- 2025Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention
- 2026Multi-Head Low-Rank Attention