mlsanityData to 7 Oct 2026

Optimised Grouped-Query Attention Mechanism for Transformers

arXiv 2406.14963 ↗First posted 21 Jun 2024cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Grouped-query attention (GQA) has been widely adopted in LLMs to mitigate the complexity of multi-head attention (MHA). To transform an MHA to a GQA, neighbour queries in MHA are evenly split into groups where each group shares the value and key layers. In this work, we propose AsymGQA, an activation-informed approach to asymmetrically grouping an MHA to a GQA for better model performance. Our AsymGQA outperforms the GQA within the same model size budget. For example, AsymGQA LLaMA-2-7B has an accuracy increase of 7.5% on MMLU compared to neighbour grouping. Our approach addresses the GQA's trade-off problem between model performance and hardware efficiency.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 5 of its 9 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

5 of 9 references.

References · 5

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  3. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQAcited by 6
  4. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  5. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizingcited by 4
  6. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Frameworkcited by 3
  7. 2025Changing Base Without Losing Pace: A GPU-Efficient Alternative to MatMul in DNNscited by 2
  8. 2024Beyond Uniform Query Distribution: Key-Driven Grouped Query Attentioncited by 1
  9. 2026Breaking the Bottlenecks: Scalable Diffusion Models for 3D Molecular Generationcited by 0
  10. 2026LLM-CoOpt: A Co-Design and Optimization Framework for Efficient LLM Inference on Heterogeneous Platformscited by 0
  11. 2026CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineeringcited by 0