mlsanityData to 7 Oct 2026

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

arXiv 2305.13245 ↗First posted 22 May 2023cs.CL cs.LGcited by 1,311

Not on the map. Why?

Cited by 365 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.

Key facts. Cited by 1,311 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

22 of 31 references.

References · 22

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2020Knowledge Distillation: A Surveycited by 1,578
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
10 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  6. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  7. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  8. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  9. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  10. 2022Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Taskscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 37; 2024: 319; 2025: 468; 2026: 487.

1,311 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Mistral 7Bcited by 3,065
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2025Large Language Diffusion Modelscited by 872
  12. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
38 more
  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  6. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  11. 2023The Falcon Series of Open Language Modelscited by 526
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  13. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  14. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  15. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  16. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  17. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  18. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  19. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  20. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  21. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  22. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  23. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  24. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  25. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  26. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  27. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  28. 2023Efficient Large Language Models: A Surveycited by 183
  29. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  30. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  31. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  32. 2026MiMo-V2-Flash Technical Reportcited by 163
  33. 2025Qwen2.5-1M Technical Reportcited by 158
  34. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  35. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  36. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  37. 2024Nemotron-4 340B Technical Reportcited by 143
  38. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142

50 most cited of 1,311 All 1,311 (JSON)