mlsanityData to 14 Oct 2026

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

arXiv 2312.07987 ↗First posted 13 Dec 2023cs.LG cs.CL cs.NEcited by 34

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Despite many recent works on Mixture of Experts (MoEs) for resource-efficient Transformer language models, existing methods mostly focus on MoEs for feedforward layers. Previous attempts at extending MoE to the self-attention layer fail to match the performance of the parameter-matched baseline. Our novel SwitchHead is an effective MoE method for the attention layer that successfully reduces both the compute and memory requirements, achieving wall-clock speedup, while matching the language modeling performance of the baseline Transformer. Our novel MoE mechanism allows SwitchHead to compute up to 8 times fewer attention matrices than the standard Transformer. SwitchHead can also be combined with MoE feedforward layers, resulting in fully-MoE "SwitchAll" Transformers. For our 262M parameter model trained on C4, SwitchHead matches the perplexity of standard models with only 44% compute and 27% memory usage. Zero-shot experiments on downstream tasks confirm the performance of SwitchHead, e.g., achieving more than 3.5% absolute improvements on BliMP compared to the baseline with an equal compute resource.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 32 of its 46 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

32 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
17 more
  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  9. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  10. 2021Random Feature Attentioncited by 354
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  12. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  13. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  14. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  15. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  16. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  17. 2020A Mixture of h-1 Heads is Better than h Headscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 15; 2026: 11.

34 papers.

Sort:

By area:

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  2. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  6. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  8. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 44
  9. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  10. 2024Scattered Mixture-of-Experts Implementationcited by 15
  11. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 14
  12. 2026Sparse Layers are Critical to Scaling Looped Language Modelscited by 11
22 more
  1. 2025Learning to Skip the Middle Layers of Transformerscited by 10
  2. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  3. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 4
  4. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  5. 2026Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Loopingcited by 3
  6. 2026MoRE: Mixture of Reused Expertscited by 3
  7. 2025BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASRcited by 2
  8. 2025Reasoning-Intensive Regressioncited by 2
  9. 2025DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networkscited by 2
  10. 2026Grouped Query Experts: Mixture-of-Experts on GQA Self-Attentioncited by 2
  11. 2025Mixture of Group Experts for Learning Invariant Representationscited by 1
  12. 2026LiMoDE: Rethinking Lifelong Robot Manipulation from a Mixture-of-Dynamic-Experts Perspectivecited by 1
  13. 2025RouterKT: Mixture-of-Experts for Knowledge Tracingcited by 0
  14. 2025Efficient Inter-Task Attention for Multitask Transformer Modelscited by 0
  15. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problemscited by 0
  16. 2025HIMOSA: Efficient Remote Sensing Image Super-Resolution with Hierarchical Mixture of Sparse Attentioncited by 0
  17. 2026Non-Interfering Weight Fields: Treating Model Parameters as a Continuously Extensible Functioncited by 0
  18. 2026Swimba: Switch Mamba Model Scales State Space Modelscited by 0
  19. 2026Transformer as an Euler Discretization of Score-based Variational Flowcited by 0
  20. 2026Gated Graph Attention Networks with Learnable Temperaturecited by 0
  21. 2026TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocationcited by 0
  22. 2026Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Headcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
  2. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  3. 2024Mixture of A Million Experts
  4. 2025Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
  5. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts
  6. 2024Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
  7. 2022Efficient Sparsely Activated Transformers
  8. 2026A theoretical model for task routing in mixture-of-expert transformers