# GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

arXiv: https://arxiv.org/abs/2605.15250 · HTML: https://mlsanity.com/paper/2605.15250/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [AI and Memory Wall](https://mlsanity.com/paper/2403.14123/) (2024)
- [Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs](https://mlsanity.com/paper/2502.14837/) (2025)
- [HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention](https://mlsanity.com/paper/2603.28458/) (2026)
- [Hardware-Efficient Attention for Fast Decoding](https://mlsanity.com/paper/2505.21487/) (2025)

## Built on by (most cited first, 1 of 1)

- [Think Before You Grid-Search: Floor-First Triage for LLM Serving](https://mlsanity.com/paper/2607.05876/) (2026)

## Abstract (arXiv)

Decoding efficiency depends on KV-cache traffic, computation, and parallelism. Multi-head Latent Attention (MLA) offers absorbed and head-expanded forms, but its head-specific projections limit grouped KV sharing. We introduce Group-Query Latent Attention (GQLA), whose group-shared up-projections preserve a joint latent and enable a shardable expanded cache. The same weights support equivalent MQA-absorb and GQA execution using existing kernels. Roofline analysis relates path preference to GPU compute-to-bandwidth ratio and tensor parallelism (TP): absorption trades traffic for computation and replicates the latent across ranks; GQA shards KV groups. Pretraining with eight query heads per group yields terminal training losses close to MLA's across the tested model scales at matched tokens. TransGQLA converts GQA and MLA checkpoints using calibration data. Without gradient updates, converted DeepSeek-V3.1-Base achieves 77.79% eight-task accuracy versus the source's 77.84%, while WikiText-2 perplexity rises from 3.21 to 6.05. On eight H20-3e GPUs, its GQA path improves output throughput over MQA-absorb by 22.42--34.97% at matched resident batches across 8K--64K inputs. Its smaller per-rank cache permits larger batches, giving 50.07% in a separate capacity-tuned comparison. These results support multiple execution paths as an architectural means of adapting decoding to hardware and parallelism.