mlsanityData to 7 Oct 2026

On the Role of Discrete Representation in Sparse Mixture of Experts

arXiv 2411.19402 ↗First posted 28 Nov 2024cs.LGcited by 4

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, responsible for directing the input to relevant experts; however, it also presents a major weakness, leading to routing inconsistencies and representation collapse issues. Instead of fixing the router like previous works, we propose an alternative that assigns experts to input via indirection, which employs the discrete representation of input that points to the expert. The discrete representations are learnt via vector quantization, resulting in a new architecture dubbed Vector-Quantized Mixture of Experts (VQMoE). We provide theoretical support and empirical evidence demonstrating the VQMoE's ability to overcome the challenges present in traditional routers. Through extensive evaluations on both large language models and vision tasks for pre-training and fine-tuning, we show that VQMoE achieves a 28% improvement in robustness compared to other SMoE routing methods, while maintaining strong performance in fine-tuning tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 36 of its 51 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

36 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023From Sparse to Soft Mixtures of Experts2 Aug
  2. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar
  3. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
21 more
  1. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  2. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  6. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  9. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  10. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  11. 2017Learning Discrete Representations via Information Maximizing Self-Augmented Trainingcited by 149
  12. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  13. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  14. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  15. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  16. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  17. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  18. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  19. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognitioncited by 26
  20. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
  21. 2024SimSMoE: Solving Representational Collapse via Similarity Measurecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Routing-Free Mixture-of-Expertscited by 2
  2. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1
  3. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  4. 2026Adaptive Inverted-Index Routing for Granular Mixtures-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  2. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  3. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
  4. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  5. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  6. 2025S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
  7. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  8. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts