# Optimised Grouped-Query Attention Mechanism for Transformers

arXiv: https://arxiv.org/abs/2406.14963 · HTML: https://mlsanity.com/paper/2406.14963/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 5 of 5)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)

## Built on by (most cited first, 12 of 12)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Palu: Compressing KV-Cache with Low-Rank Projection](https://mlsanity.com/paper/2407.21118/) (2024)
- [Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA](https://mlsanity.com/paper/2412.20677/) (2024)
- [SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention](https://mlsanity.com/paper/2502.15304/) (2025)
- [MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing](https://mlsanity.com/paper/2506.02006/) (2025)
- [FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework](https://mlsanity.com/paper/2503.08461/) (2025)
- [Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention](https://mlsanity.com/paper/2408.08454/) (2024)
- [Changing Base Without Losing Pace: A GPU-Efficient Alternative to MatMul in DNNs](https://mlsanity.com/paper/2503.12211/) (2025)
- [Opt-GPTQ: An Optimized GPTQ Combining Sparse Attention and Quantization Techniques](https://mlsanity.com/paper/2505.02351/) (2025)
- [Breaking the Bottlenecks: Scalable Diffusion Models for 3D Molecular Generation](https://mlsanity.com/paper/2601.08963/) (2026)
- [LLM-CoOpt: A Co-Design and Optimization Framework for Efficient LLM Inference on Heterogeneous Platforms](https://mlsanity.com/paper/2602.09323/) (2026)
- [CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineering](https://mlsanity.com/paper/2610.09476/) (2026)

## Abstract (arXiv)

Grouped-query attention (GQA) has been widely adopted in LLMs to mitigate the complexity of multi-head attention (MHA). To transform an MHA to a GQA, neighbour queries in MHA are evenly split into groups where each group shares the value and key layers. In this work, we propose AsymGQA, an activation-informed approach to asymmetrically grouping an MHA to a GQA for better model performance. Our AsymGQA outperforms the GQA within the same model size budget. For example, AsymGQA LLaMA-2-7B has an accuracy increase of 7.5% on MMLU compared to neighbour grouping. Our approach addresses the GQA's trade-off problem between model performance and hardware efficiency.