mlsanityData to 7 Oct 2026

On the Representation Collapse of Sparse Mixture of Experts

arXiv 2204.09179 ↗First posted 20 Apr 2022cs.CL cs.LGcited by 156

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations. However, learning such a routing mechanism encourages token clustering around expert centroids, implying a trend toward representation collapse. In this work, we propose to estimate the routing scores between tokens and experts on a low-dimensional hypersphere. We conduct extensive experiments on cross-lingual language model pre-training and fine-tuning on downstream tasks. Experimental results across seven multilingual benchmarks show that our method achieves consistent gains. We also present a comprehensive analysis on the representation and routing behaviors of our models. Our method alleviates the representation collapse issue and achieves more consistent routing than the baseline mixture-of-experts methods.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 40 of its 50 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

40 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  2. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
25 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  9. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  10. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  12. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  13. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  14. 2020Whitening for Self-Supervised Representation Learningcited by 278
  15. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  16. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  17. 2021Hash Layers For Large Sparse Modelscited by 236
  18. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  19. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  20. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  21. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  22. 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
  23. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  24. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  25. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 20; 2024: 42; 2025: 28; 2026: 57.

156 papers.

Sort:

By area:

  1. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  2. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  3. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  6. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  7. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  8. 2022Language Models are General-Purpose Interfacescited by 96
  9. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  10. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  11. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  12. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
38 more
  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  2. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  5. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  6. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  7. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  8. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  9. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  10. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  11. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  12. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  13. 2024A Survey on Transformer Compressioncited by 44
  14. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42
  15. 2022VL-BEiT: Generative Vision-Language Pretrainingcited by 39
  16. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  17. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attentioncited by 34
  18. 2024Multi-Head Mixture-of-Expertscited by 30
  19. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  20. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  21. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  22. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  23. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  24. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  25. 2023DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasetscited by 23
  26. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  27. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  28. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  29. 2024Ultra-Sparse Memory Networkcited by 19
  30. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  31. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  32. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  33. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  34. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  35. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  36. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 15
  37. 2022MoEC: Mixture of Expert Clusterscited by 14
  38. 2024A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Expertscited by 13

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts
  2. 2026Mixture of Experts for Low-Resource LLMs
  3. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  4. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  5. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  6. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  7. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  8. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference