mlsanityData to 7 Oct 2026

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT

arXiv 2205.12399 ↗First posted 24 May 2022cs.LG cs.CLcited by 12

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. Sparse Mixer slightly outperforms (<1%) BERT on GLUE and SuperGLUE, but more importantly trains 65% faster and runs inference 61% faster. We also present a faster variant, prosaically named Fast Sparse Mixer, that marginally underperforms BERT on SuperGLUE, but trains and runs nearly twice as fast. We justify the design of these two models by carefully ablating through various mixing mechanisms, MoE configurations and hyperparameters. Sparse Mixer overcomes many of the latency and stability concerns of MoE models and offers the prospect of serving sparse student models, without resorting to distilling them to dense variants.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 61 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

42 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
26 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2020GLU Variants Improve Transformercited by 1,724
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2021Pay Attention to MLPscited by 367
  13. 2014Memory Networkscited by 354
  14. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  15. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  16. 2021Hash Layers For Large Sparse Modelscited by 236
  17. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  18. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  19. 2019Large Memory Layers with Product Keyscited by 120
  20. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  21. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  22. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  23. 2021Sparse is Enough in Scaling Transformerscited by 86
  24. 2020Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translationcited by 42
  25. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  26. 2021PyGlove: Symbolic Programming for Automated Machine Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 6; 2025: 0; 2026: 1.

12 papers.

  1. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  3. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  5. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  6. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  7. 2024Training-efficient density quantum machine learningcited by 10
  8. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  9. 2023G-SPEED: General SParse Efficient Editing MoDelcited by 0
  10. 2023Direct Neural Machine Translation with Task-level Mixture of Experts modelscited by 0
  11. 2023Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transformcited by 0
  12. 2026B-MoE: A Body-Part-Aware Mixture-of-Experts "All Parts Matter" Approach to Micro-Action Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  3. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  4. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  5. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  7. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  8. 2026Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution