mlsanityData to 7 Oct 2026

OLMoE: Open Mixture-of-Experts Language Models

arXiv 2409.02060 ↗First posted 3 Sep 2024cs.CL cs.AI cs.LGcited by 291

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs.

Key facts. Cited by 291 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 200 of its 223 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

200 of 223 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts15 Aug
  2. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  4. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training6 May
  5. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models8 Apr
  7. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models3 Apr
  8. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  9. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM12 Mar
  10. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  11. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
8 more
  1. 2024Mixtral of Experts8 Jan
  2. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models24 May
  3. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 180

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2016Layer Normalizationcited by 5,579
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023Qwen Technical Reportcited by 3,273
  13. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  14. 2023Mistral 7Bcited by 3,065
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  16. 2016Pointer Sentinel Mixture Modelscited by 2,992
  17. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  18. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  19. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  20. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  21. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  22. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  23. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  24. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  25. 2024Qwen2 Technical Reportcited by 2,076
  26. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  27. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  28. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  29. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  30. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  31. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  32. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  33. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  34. 2020GLU Variants Improve Transformercited by 1,724
  35. 2017Decoupled Weight Decay Regularizationcited by 1,722
  36. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  37. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  38. 2019Root Mean Square Layer Normalizationcited by 1,425

50 most cited of 180 All 180 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 110; 2026: 154.

291 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  3. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  4. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  5. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  6. 2024Scaling Laws for Precisioncited by 92
  7. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  8. 2025Energy Considerations of Large Language Model Inference and Efficiency Optimizationscited by 54
  9. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  10. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  11. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  12. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
38 more
  1. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 46
  2. 2025Diffusion Language Models are Super Data Learnerscited by 45
  3. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  4. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  5. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  6. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  7. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  8. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  9. 2025OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokenscited by 27
  10. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  11. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  12. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  13. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  14. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  15. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  16. 2025Training Optimal Large Diffusion Language Modelscited by 21
  17. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  18. 2025Enough Coin Flips Can Make LLMs Act Bayesiancited by 19
  19. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  20. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  21. 2025Two Experts Are All You Need for Steering Thinking: Reinforcing Cognitive Effort in MoE Reasoning Models Without Additional Trainingcited by 16
  22. 2025Intrinsic Fingerprint of LLMs: Continue Training is NOT All You Need to Steal A Model!cited by 16
  23. 2025HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalizationcited by 15
  24. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 14
  25. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  26. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  27. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  28. 2024LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluationcited by 12
  29. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 12
  30. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  31. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 12
  32. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 12
  33. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 12
  34. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  35. 2025Mixture of Lookup Expertscited by 11
  36. 2025UCCL-EP: Portable Expert-Parallel Communicationcited by 11
  37. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  38. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9

50 most cited of 291 All 291 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  2. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  3. 2025Mixture of Experts in Large Language Models
  4. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  5. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  7. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
  8. 2026MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models