mlsanityData to 7 Oct 2026

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs

arXiv 2502.14837 ↗First posted 20 Feb 2025cs.CL cs.AIcited by 27

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Multi-head Latent Attention (MLA) is an innovative architecture proposed by DeepSeek, designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector. Compared to MLA, standard LLMs employing Multi-Head Attention (MHA) and its variants such as Grouped-Query Attention (GQA) exhibit significant cost disadvantages. Enabling well-trained LLMs (e.g., Llama) to rapidly adapt to MLA without pre-training from scratch is both meaningful and challenging. This paper proposes the first data-efficient fine-tuning method for transitioning from MHA to MLA (MHA2MLA), which includes two key components: for partial-RoPE, we remove RoPE from dimensions of queries and keys that contribute less to the attention scores, for low-rank approximation, we introduce joint SVD approximations based on the pre-trained parameters of keys and values. These carefully designed strategies enable MHA2MLA to recover performance using only a small fraction (0.3% to 0.6%) of the data, significantly reducing inference costs while seamlessly integrating with compression techniques such as KV cache quantization. For example, the KV cache size of Llama2-7B is reduced by 92.19%, with only a 0.5% drop in LongBench performance.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 26 of its 33 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

26 of 33 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2020Rethinking Attention with Performerscited by 1,584
14 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  10. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  11. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  12. 2019Star-Transformercited by 94
  13. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 20
  14. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decodercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 14.

27 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  3. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  4. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 7
  5. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modelingcited by 7
  6. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUscited by 5
  7. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
  8. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compressioncited by 4
  9. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  10. 2025Controlling changes to attention logitscited by 3
  11. 2026Generalizable Foundation Models for Calorimetry via Mixtures-of-Experts and Parameter Efficient Fine Tuningcited by 3
  12. 2026VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusioncited by 3
15 more
  1. 2025DoPE: Denoising Rotary Position Embeddingcited by 2
  2. 2026Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversioncited by 2
  3. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
  4. 2026Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversioncited by 2
  5. 2026GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decodingcited by 1
  6. 2026SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Codingcited by 1
  7. 2025Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulationcited by 0
  8. 2025Understanding Differential Transformer Unchains Pretrained Self-Attentionscited by 0
  9. 2025EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMscited by 0
  10. 2025OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latencycited by 0
  11. 2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Modelscited by 0
  12. 2026YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transitioncited by 0
  13. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Cachingcited by 0
  14. 2026SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attentioncited by 0
  15. 2026Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA
  2. 2025TransMLA: Multi-Head Latent Attention Is All You Need
  3. 2026Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
  4. 2025Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention
  5. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
  6. 2024CHAI: Clustered Head Attention for Efficient LLM Inference
  7. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
  8. 2025Latent Multi-Head Attention for Small Language Models