mlsanityData to 7 Oct 2026

MoH: Multi-Head Attention as Mixture-of-Head Attention

arXiv 2410.11842 ↗First posted 15 Oct 2024cs.CV cs.AI cs.LGcited by 41

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

In this work, we upgrade the multi-head attention mechanism, the core of the Transformer model, to improve efficiency while maintaining or surpassing the previous accuracy level. We show that multi-head attention can be expressed in the summation form. Drawing on the insight that not all attention heads hold equal significance, we propose Mixture-of-Head attention (MoH), a new architecture that treats attention heads as experts in the Mixture-of-Experts (MoE) mechanism. MoH has two significant advantages: First, MoH enables each token to select the appropriate attention heads, enhancing inference efficiency without compromising accuracy or increasing the number of parameters. Second, MoH replaces the standard summation in multi-head attention with a weighted summation, introducing flexibility to the attention mechanism and unlocking extra performance potential. Extensive experiments on ViT, DiT, and LLMs demonstrate that MoH outperforms multi-head attention by using only 50%-90% of the attention heads. Moreover, we demonstrate that pre-trained multi-head attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14 benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the attention heads. We believe the proposed MoH is a promising alternative to multi-head attention and provides a strong foundation for developing advanced and efficient attention-based models.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 88 of its 103 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

88 of 103 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention13 Dec
  2. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2024The Llama 3 Herd of Modelscited by 14,192
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  13. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  14. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  15. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  16. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  17. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  18. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  19. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  20. 2024Mixtral of Expertscited by 1,785
  21. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  22. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  23. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  24. 2017Random Erasing Data Augmentationcited by 1,235
  25. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  26. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  27. 2016Deep Networks with Stochastic Depthcited by 973
  28. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  29. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  30. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  31. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  32. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  33. 2019Are Sixteen Heads Really Better than One?cited by 803
  34. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  35. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  36. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  37. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  38. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 18.

41 papers.

Sort:

By area:

  1. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  4. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  5. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  6. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  7. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  8. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  9. 2025GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretabilitycited by 9
  10. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  11. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  12. 2025Route Experts by Sequence, not by Tokencited by 5
29 more
  1. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inferencecited by 5
  2. 2025Safe-VAR: Safe Visual Autoregressive Model for Text-to-Image Generative Watermarkingcited by 4
  3. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 4
  4. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
  5. 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
  6. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learningcited by 3
  7. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  8. 2026Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Loopingcited by 3
  9. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Informationcited by 3
  10. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  11. 2025HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detectioncited by 2
  12. 2026Making Training-Free Diffusion Segmentors Scale with the Generative Powercited by 2
  13. 2026A2-Edit: Precise Reference-Guided Image Editing of Arbitrary Objects and Ambiguous Maskscited by 2
  14. 2026Grouped Query Experts: Mixture-of-Experts on GQA Self-Attentioncited by 2
  15. 2026LoSA-Net: A Localized and Scale-Adaptive Network for Boundary-Sensitive Prediction of Perineural Invasion in 3D MRIcited by 2
  16. 2025Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimizationcited by 1
  17. 2025Knocking-Heads Attentioncited by 1
  18. 2026Improving MoE Compute Efficiency by Composing Weight and Data Sparsitycited by 1
  19. 2025RouterKT: Mixture-of-Experts for Knowledge Tracingcited by 0
  20. 2025Efficient Mathematical Reasoning Models via Dynamic Pruning and Knowledge Distillationcited by 0
  21. 2026M2FMoE: Multi-Resolution Multi-View Frequency Mixture-of-Experts for Extreme-Adaptive Time Series Forecastingcited by 0
  22. 2026Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generationcited by 0
  23. 2026Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformerscited by 0
  24. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  25. 2026MMA-Former: Multi-Window Mixture-of-Head Attention Transformer for Adaptive PNI Prediction in 3D MRIcited by 0
  26. 2026VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformerscited by 0
  27. 2026Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Headcited by 0
  28. 2026ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Controlcited by 0
  29. 2026Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contextscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
  2. 2026Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
  3. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
  4. 2026Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMs
  6. 2026Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
  7. 2024CHAI: Clustered Head Attention for Efficient LLM Inference
  8. 2026Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights