# A Closer Look into Mixture-of-Experts in Large Language Models

arXiv: https://arxiv.org/abs/2406.18219 · HTML: https://mlsanity.com/paper/2406.18219/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 25 of 25)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://mlsanity.com/paper/2212.05055/) (2022)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [On the Representation Collapse of Sparse Mixture of Experts](https://mlsanity.com/paper/2204.09179/) (2022)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [Taming Sparsely Activated Transformer with Stochastic Experts](https://mlsanity.com/paper/2110.04260/) (2021)
- [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023)
- [StableMoE: Stable Routing Strategy for Mixture of Experts](https://mlsanity.com/paper/2204.08396/) (2022)
- [NormFormer: Improved Transformer Pretraining with Extra Normalization](https://mlsanity.com/paper/2110.09456/) (2021)
- [Mixture of Attention Heads: Selecting Attention Heads Per Token](https://mlsanity.com/paper/2210.05144/) (2022)
- [Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models](https://mlsanity.com/paper/2406.06563/) (2024)
- [Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters](https://mlsanity.com/paper/2406.05955/) (2024)
- [CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition](https://mlsanity.com/paper/2402.02526/) (2024)
- [Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast](https://mlsanity.com/paper/2405.14507/) (2024)
- [Yuan 2.0-M32: Mixture of Experts with Attention Router](https://mlsanity.com/paper/2405.17976/) (2024)
- [Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers](https://mlsanity.com/paper/2402.12233/) (2024)

## Built on by (most cited first, 30 of 48)

- [Multilingual Routing in Mixture-of-Experts](https://mlsanity.com/paper/2510.04694/) (2025)
- [Steering MoE LLMs via Expert (De)Activation](https://mlsanity.com/paper/2509.09660/) (2025)
- [Layerwise Recurrent Router for Mixture-of-Experts](https://mlsanity.com/paper/2408.06793/) (2024)
- [Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts](https://mlsanity.com/paper/2503.05066/) (2025)
- [Scaling Laws for Upcycling Mixture-of-Experts Language Models](https://mlsanity.com/paper/2502.03009/) (2025)
- [NeurIPT: Foundation Model for Neural Interfaces](https://mlsanity.com/paper/2510.16548/) (2025)
- [Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Test](https://mlsanity.com/paper/2506.21551/) (2025)
- [PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model](https://mlsanity.com/paper/2411.08212/) (2024)
- [Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts](https://mlsanity.com/paper/2604.08541/) (2026)
- [Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models](https://mlsanity.com/paper/2505.16056/) (2025)
- [Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource](https://mlsanity.com/paper/2506.12119/) (2025)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [MoE Pathfinder: Trajectory-driven Expert Pruning](https://mlsanity.com/paper/2512.18425/) (2025)
- [The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level](https://mlsanity.com/paper/2604.02178/) (2026)
- [Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging](https://mlsanity.com/paper/2410.01610/) (2024)
- [The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models](https://mlsanity.com/paper/2601.03425/) (2026)
- [Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency](https://mlsanity.com/paper/2603.17102/) (2026)
- [Mixture-of-Shape-Experts (MoSE): End-to-End Shape Dictionary Framework to Prompt SAM for Generalizable Medical Segmentation](https://mlsanity.com/paper/2504.09601/) (2025)
- [From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan](https://mlsanity.com/paper/2507.09205/) (2025)
- [Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.12476/) (2026)
- [Multilingual State Space Models for Structured Question Answering in Indic Languages](https://mlsanity.com/paper/2502.01673/) (2025)
- [CoSMoEs: Compact Sparse Mixture of Experts](https://mlsanity.com/paper/2503.00245/) (2025)
- [Expert Merging in Sparse Mixture of Experts with Nash Bargaining](https://mlsanity.com/paper/2510.16138/) (2025)
- [What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models](https://mlsanity.com/paper/2601.10159/) (2026)
- [Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts](https://mlsanity.com/paper/2605.28042/) (2026)
- [Mixture-of-Experts for Personalized and Semantic-Aware Next Location Prediction](https://mlsanity.com/paper/2505.24597/) (2025)
- [Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations](https://mlsanity.com/paper/2601.00457/) (2026)
- [Compressed Sensing for Capability Localization in Large Language Models](https://mlsanity.com/paper/2603.03335/) (2026)
- [Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation](https://mlsanity.com/paper/2605.29714/) (2026)
- [Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations](https://mlsanity.com/paper/2607.20426/) (2026)

## Abstract (arXiv)

Mixture-of-experts (MoE) is gaining increasing attention due to its unique properties and remarkable performance, especially for language tasks. By sparsely activating a subset of parameters for each token, MoE architecture could increase the model size without sacrificing computational efficiency, achieving a better trade-off between performance and training costs. However, the underlying mechanism of MoE still lacks further exploration, and its modularization degree remains questionable. In this paper, we make an initial attempt to understand the inner workings of MoE-based large language models. Concretely, we comprehensively study the parametric and behavioral features of three popular MoE-based models and reveal some intriguing observations, including 1) Neurons act like fine-grained experts; 2) The router of MoE usually selects experts with larger output norms; 3) The expert diversity increases as the layer increases, while the last layer is an outlier, which is further validated by an initial experiment. Based on the observations, we also provide suggestions for a broad spectrum of MoE practitioners, such as router design and expert allocation. We hope this work could shed light on future research on the MoE framework and other modular architectures. Code is available at https://github.com/kamanphoebe/Look-into-MoEs.