mlsanityData to 7 Oct 2026

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

arXiv 2402.01739 ↗First posted 29 Jan 2024cs.CL cs.AI cs.DCcited by 185

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens. Our investigation confirms that MoE-based LLMs can offer a more favorable cost-effectiveness trade-off than dense LLMs, highlighting the potential effectiveness for future LLM development. One more important contribution of this study is an in-depth analysis of the routing mechanisms within our OpenMoE models, leading to three significant findings: Context-Independent Specialization, Early Routing Learning, and Drop-towards-the-End. We discovered that routing decisions in MoE models are predominantly based on token IDs, with minimal context relevance. The token-to-expert assignments are determined early in the pre-training phase and remain largely unchanged. This imperfect routing can result in performance degradation, particularly in sequential tasks like multi-turn conversations, where tokens appearing later in a sequence are more likely to be dropped. Finally, we rethink our design based on the above-mentioned observations and analysis. To facilitate future MoE LLM development, we propose potential strategies for mitigating the issues we found and further improving off-the-shelf MoE LLM designs.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 50 of its 61 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

50 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2024Mixtral of Experts8 Jan
  3. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  4. 2022Efficient Language Modeling with Sparse all-MLP14 Mar
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
29 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2023StarCoder: may the source be with you!cited by 790
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  13. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  14. 2021Deep Long-Tailed Learning: A Surveycited by 416
  15. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  16. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  17. 2022UL2: Unifying Language Learning Paradigmscited by 292
  18. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  19. 2023From Sparse to Soft Mixtures of Expertscited by 245
  20. 2021Hash Layers For Large Sparse Modelscited by 236
  21. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  22. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  23. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  24. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  25. 2021Go Wider Instead of Deepercited by 56
  26. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  27. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  28. 2021Cross-token Modeling with Conditional Computationcited by 17
  29. 2023XGen-7B Technical Reportcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 72; 2026: 56.

185 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  4. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  5. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  7. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  8. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  9. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  10. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  11. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  12. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
38 more
  1. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  2. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  3. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  4. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  5. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  6. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  7. 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justificationscited by 41
  8. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  9. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 38
  10. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  11. 2025Advancing Expert Specialization for Better MoEcited by 37
  12. 2024Adaptive Multi-Scale Decomposition Framework for Time Series Forecastingcited by 34
  13. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  14. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  15. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  16. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  17. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  18. 2025NTIRE 2025 Challenge on Cross-Domain Few-Shot Object Detection: Methods and Resultscited by 25
  19. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  20. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  21. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  22. 2025DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformerscited by 23
  23. 2025Steering MoE LLMs via Expert (De)Activationcited by 23
  24. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  25. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  26. 2024Towards an empirical understanding of MoE design choicescited by 20
  27. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  28. 2025MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Expertscited by 19
  29. 2024EEGMamba: Bidirectional State Space Model with Mixture of Experts for EEG Multi-task Classificationcited by 18
  30. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 18
  31. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  32. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  33. 2025Mid-Training of Large Language Models: A Surveycited by 18
  34. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  35. 2025Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Trainingcited by 16
  36. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 16
  37. 2024Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Expertscited by 14
  38. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  2. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
  3. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference
  4. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  5. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  6. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  7. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  8. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts