mlsanityData to 14 Oct 2026

TransMLA: Multi-Head Latent Attention Is All You Need

arXiv 2502.07864 ↗First posted 11 Feb 2025cs.LG cs.AIcited by 23

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 41 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

26 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder30 Jul
  3. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
9 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  4. 2024Phi-4 Technical Reportcited by 711
  5. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  6. 2024Mixture of LoRA Expertscited by 181
  7. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  8. 2025Tensor Product Attention Is All You Needcited by 31
  9. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 25
  2. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  3. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 6
  4. 2026VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusioncited by 6
  5. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  6. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attentioncited by 5
  7. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancingcited by 4
  8. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstructioncited by 4
  9. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 3
  10. 2025xLLM Technical Reportcited by 3
  11. 2025Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Modelscited by 3
  12. 2026Tucker Attention: A generalization of approximate attention mechanismscited by 3
11 more
  1. 2025Latent Multi-Head Attention for Small Language Modelscited by 2
  2. 2026Multi-Head Low-Rank Attentioncited by 2
  3. 2026Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversioncited by 2
  4. 2025A Random Matrix Theory Perspective on the Learning Dynamics of Multi-head Latent Attentioncited by 1
  5. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  6. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Servingcited by 1
  7. 2026The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architecturescited by 1
  8. 2026SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Codingcited by 1
  9. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 1
  10. 2026The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendscited by 0
  11. 2026Hybrid Latent Attention for Looped Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
  2. 2026MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs
  3. 2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
  4. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs
  5. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
  6. 2026GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
  7. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
  8. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA