mlsanityData to 7 Oct 2026

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

arXiv 2606.20945 ↗First posted 18 Jun 2026cs.LGcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 18 of its 18 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

18 of 18 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MoH: Multi-Head Attention as Mixture-of-Head Attention15 Oct
  2. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention13 Dec
  3. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 14

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
  11. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  12. 2020A Mixture of h-1 Heads is Better than h Headscited by 22
2 more
  1. 2024Scattered Mixture-of-Experts Implementationcited by 15
  2. 2025ZClip: Adaptive Spike Mitigation for LLM Pre-Trainingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstructioncited by 0
  2. 2026Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Headcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
  3. 2026Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights
  4. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
  5. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  6. 2026MiniMax Sparse Attention
  7. 2025Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
  8. 2024Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA