mlsanityData to 14 Oct 2026

A Closer Look into Mixture-of-Experts in Large Language Models

arXiv 2406.18219 ↗First posted 26 Jun 2024cs.CL cs.LGcited by 48

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-experts (MoE) is gaining increasing attention due to its unique properties and remarkable performance, especially for language tasks. By sparsely activating a subset of parameters for each token, MoE architecture could increase the model size without sacrificing computational efficiency, achieving a better trade-off between performance and training costs. However, the underlying mechanism of MoE still lacks further exploration, and its modularization degree remains questionable. In this paper, we make an initial attempt to understand the inner workings of MoE-based large language models. Concretely, we comprehensively study the parametric and behavioral features of three popular MoE-based models and reveal some intriguing observations, including 1) Neurons act like fine-grained experts; 2) The router of MoE usually selects experts with larger output norms; 3) The expert diversity increases as the layer increases, while the last layer is an outlier, which is further validated by an initial experiment. Based on the observations, we also provide suggestions for a broad spectrum of MoE practitioners, such as router design and expert allocation. We hope this work could shed light on future research on the MoE framework and other modular architectures. Code is available at https://github.com/kamanphoebe/Look-into-MoEs.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 25 of its 38 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

25 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters10 Jun
  2. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  3. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  7. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  8. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  9. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  10. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  11. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  12. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
6 more
  1. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  2. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 60
  3. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  4. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  5. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18
  6. 2024Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layerscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 27.

48 papers.

Sort:

By area:

  1. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  2. 2025Steering MoE LLMs via Expert (De)Activationcited by 26
  3. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  4. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 15
  5. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  6. 2025NeurIPT: Foundation Model for Neural Interfacescited by 11
  7. 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Testcited by 10
  8. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  9. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 9
  10. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  11. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 8
  12. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
36 more
  1. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  2. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  3. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
  4. 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Modelscited by 6
  5. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  6. 2025Mixture-of-Shape-Experts (MoSE): End-to-End Shape Dictionary Framework to Prompt SAM for Generalizable Medical Segmentationcited by 4
  7. 2025From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetancited by 4
  8. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Expertscited by 4
  9. 2025Multilingual State Space Models for Structured Question Answering in Indic Languagescited by 3
  10. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
  11. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargainingcited by 3
  12. 2026What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Modelscited by 3
  13. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  14. 2025Mixture-of-Experts for Personalized and Semantic-Aware Next Location Predictioncited by 2
  15. 2026Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activationscited by 2
  16. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  17. 2026Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptationcited by 2
  18. 2026Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinationscited by 2
  19. 2024Spatio-Temporal Multi-Subgraph GCN for 3D Human Motion Predictioncited by 1
  20. 2025Three tiers of computation in transformers and in brain architecturescited by 1
  21. 2025MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Securitycited by 1
  22. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  23. 2026Sparse Multi-Stage Expert-Agent Routing for Complex Clinical Reasoningcited by 1
  24. 2026A Closer Look into LLMs for Table Understandingcited by 0
  25. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  26. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  27. 2026Exploring Concreteness Through a Figurative Lenscited by 0
  28. 2026Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Expertscited by 0
  29. 2026Decomposing Refusal Steering in Mixture-of-Experts Modelscited by 0
  30. 2026From Black-Box to Clinical Insight: A Multi-Stage Explainable Framework for Speech-Based Cognitive Impairment Detectioncited by 0
  31. 2026Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?cited by 0
  32. 2026Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Modelscited by 0
  33. 2026Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thoughtcited by 0
  34. 2026CORTEX: Learning to Share and Specialize in Dense Language Modelscited by 0
  35. 2026From Routing Signals to Selective Review: Visual regrounding in MoE VLMscited by 0
  36. 2026InstMoE: Adaptive Multimodal Routing with Specialized Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  3. 2024A Survey on Mixture of Experts in Large Language Models
  4. 2025Advancing Expert Specialization for Better MoE
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  6. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  7. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  8. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models