mlsanityData to 7 Oct 2026

Multi-Head Low-Rank Attention

arXiv 2603.02188 ↗First posted 2 Mar 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context inference in large language models is bottlenecked by Key--Value (KV) cache loading during the decoding stage, where the sequential nature of generation requires repeatedly transferring the KV cache from off-chip High-Bandwidth Memory (HBM) to on-chip Static Random-Access Memory (SRAM) at each step. While Multi-Head Latent Attention (MLA) significantly reduces the total KV cache size, it suffers from a sharding bottleneck during distributed decoding via Tensor Parallelism (TP). Since its single latent head cannot be partitioned, each device is forced to redundantly load the complete KV cache for every token, consuming excessive memory traffic and diminishing TP benefits like weight sharding. In this work, we propose Multi-Head Low-Rank Attention (MLRA), which enables partitionable latent states for efficient 4-way TP decoding. Extensive experiments show that MLRA achieves state-of-the-art perplexity and downstream task performance, while also delivering a 2.8× decoding speedup over MLA. Code is available at https://github.com/SongtaoLiu0823/MLRA. Pretrained weights, along with the training and evaluation data, are available at https://huggingface.co/Soughing/MLRA.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 83 of its 100 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

83 of 100 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Hardware-Efficient Attention for Fast Decoding27 May
  2. 2025TransMLA: Multi-Head Latent Attention Is All You Need11 Feb
  3. 2024Multi-matrix Factorization Attention26 Dec
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  6. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024LoCoCo: Dropping In Convolutions for Long Context Compression8 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
3 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  13. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  14. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  15. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  16. 2021E(n) Equivariant Graph Neural Networkscited by 951
  17. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  18. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  19. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  20. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  21. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  22. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  23. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  24. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  25. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  26. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  27. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  28. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  29. 2021Random Feature Attentioncited by 350
  30. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  31. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  32. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  33. 2015Highway Networkscited by 257
  34. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
  35. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  36. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  37. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  38. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
  2. 2026GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding
  3. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decoding
  4. 2024Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
  5. 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attention
  6. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  7. 2025EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
  8. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs