mlsanityData to 7 Oct 2026

SimSMoE: Solving Representational Collapse via Similarity Measure

arXiv 2406.15883 ↗First posted 22 Jun 2024cs.CL cs.AIcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse mixture of experts (SMoE) have emerged as an effective approach for scaling large language models while keeping a constant computational cost. Regardless of several notable successes of SMoE, effective training such architecture remains elusive due to the representation collapse problem, which in turn harms model performance and causes parameter redundancy. In this work, we present Similarity-based Sparse Mixture of Experts (SimSMoE), a novel similarity of neural network algorithm, that guarantees a solution to address the representation collapse issue between experts given a fixed FLOPs budget. We conduct extensive empirical evaluations on three large language models for both Pre-training and Fine-tuning tasks to illustrate the efficacy, robustness, and scalability of our method. The results demonstrate that SimSMoE significantly enhances existing routing policy and outperforms other SMoE training methods in performance for the tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 51 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

34 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models24 May
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
18 more
  1. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  2. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  3. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  4. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  5. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  6. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  7. 2021Similarity and Matching of Neural Network Representationscited by 77
  8. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  9. 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
  10. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  11. 2018Geometry Score: A Method For Comparing Generative Adversarial Networkscited by 56
  12. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  13. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  14. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  15. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  16. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognitioncited by 26
  17. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
  18. 2023Revisiting the Evaluation of Image Synthesis with GANscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 6
  2. 2024On the Role of Discrete Representation in Sparse Mixture of Expertscited by 4
  3. 2026Do Domain-specific Experts exist in MoE-based LLMs?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
  2. 2025S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
  3. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  4. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
  5. 2025CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition
  6. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  7. 2026Eigenvectors of Experts are Training-free Non-collapsing Routers
  8. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning