mlsanityData to 7 Oct 2026

Mixture of Attention Heads: Selecting Attention Heads Per Token

arXiv 2210.05144 ↗First posted 11 Oct 2022cs.CLcited by 58

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing. However, the study of MoE components mostly focused on the feedforward layer in Transformer architecture. This paper proposes the Mixture of Attention Heads (MoA), a new architecture that combines multi-head attention with the MoE mechanism. MoA includes a set of attention heads that each has its own set of parameters. Given an input, a router dynamically selects a subset of k attention heads per token. This conditional computation schema allows MoA to achieve stronger performance than the standard multi-head attention layer. Furthermore, the sparsely gated MoA can easily scale up the number of attention heads and the number of parameters while preserving computational efficiency. In addition to the performance improvements, MoA also automatically differentiates heads' utilities, providing a new perspective to discuss the model's interpretability. We conducted experiments on several important tasks, including Machine Translation and Masked Language Modeling. Experiments have shown promising results on several tasks against strong baselines that involve large and very deep models.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 26 of its 36 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

26 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
12 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  4. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  5. 2020Understanding the Difficulty of Training Transformerscited by 210
  6. 2018Scaling Neural Machine Translationcited by 204
  7. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  8. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  9. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  10. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  11. 2020A Mixture of h-1 Heads is Better than h Headscited by 22
  12. 2021Tricks for Training Sparse Translation Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 17; 2025: 25; 2026: 8.

58 papers.

Sort:

By area:

  1. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  2. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  3. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  7. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  8. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  9. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
  10. 2023Sparse Universal Transformercited by 28
  11. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  12. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
38 more
  1. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  2. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  3. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  4. 2025RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretationcited by 16
  5. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learnerscited by 15
  6. 2024Scattered Mixture-of-Experts Implementationcited by 15
  7. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 15
  8. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  9. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  10. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  11. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  12. 2024Conditional computation in neural networks: principles and research trendscited by 10
  13. 2024Improving Transformers with Dynamically Composable Multi-Head Attentioncited by 8
  14. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  15. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  16. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  17. 2025BTS: Harmonizing Specialized Experts into a Generalist LLMcited by 6
  18. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  19. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  20. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  21. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
  22. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inferencecited by 5
  23. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  24. 2025VA-AR: Learning Velocity-Aware Action Representations with Mixture of Window Attentioncited by 4
  25. 2025Devil is in the Uniformity: Exploring Diverse Learners within Transformer for Image Restorationcited by 4
  26. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  27. 2025Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Trainingcited by 4
  28. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
  29. 2025MambaMoE: Mixture-of-Spectral-Spatial-Experts State Space Model for Hyperspectral Image Classificationcited by 3
  30. 2025Action is All You Need: Dual-Flow Generative Ranking Network for Recommendationcited by 3
  31. 2025Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learningcited by 3
  32. 2026Grouped Query Experts: Mixture-of-Experts on GQA Self-Attentioncited by 2
  33. 2022EIT: Enhanced Interactive Transformercited by 1
  34. 2025Breaking Thought Patterns: A Multi-Dimensional Reasoning Framework for LLMscited by 1
  35. 2025Learning When Not to Attend Globallycited by 1
  36. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representationscited by 1
  37. 2026LoGo: Token-Level Dynamic Local-Global Attentioncited by 1
  38. 2025RouterKT: Mixture-of-Experts for Knowledge Tracingcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture of Layers with Hybrid Attention
  2. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
  3. 2026Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
  4. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
  5. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
  6. 2026Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
  7. 2025Mixture of Experts in Large Language Models
  8. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design