mlsanityData to 7 Oct 2026

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

arXiv 2406.16554 ↗First posted 24 Jun 2024cs.CLcited by 122

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual Pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters. The source codes and models are available at https://github.com/pjlab-sys4nlp/llama-moe .

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 58 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

38 of 58 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Experts17 Jun
  2. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts17 Jun
  3. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  5. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  6. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation15 Apr
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2020GLU Variants Improve Transformercited by 1,724
17 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  7. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  8. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  9. 2021Hash Layers For Large Sparse Modelscited by 236
  10. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  11. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  12. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  13. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 132
  14. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  15. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  16. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  17. 2024Timo: Towards Better Temporal Reasoning for Language Modelscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 67; 2026: 33.

122 papers.

Sort:

By area:

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  2. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 88
  3. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  4. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  6. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  7. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  8. 2025Advancing Expert Specialization for Better MoEcited by 37
  9. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  10. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 28
  11. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  12. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
38 more
  1. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignmentcited by 22
  2. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  3. 2025LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenescited by 19
  4. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  5. 2025MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Expertscited by 19
  6. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 17
  7. 2025What You Have is What You Track: Adaptive and Robust Multimodal Trackingcited by 15
  8. 2024SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?cited by 14
  9. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  10. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 14
  11. 2024H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMscited by 11
  12. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  13. 2024MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarioscited by 10
  14. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruningcited by 10
  15. 2025HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policiescited by 10
  16. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  17. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  18. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  19. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 9
  20. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  21. 2025Speculate, then Collaborate: Fusing Knowledge of Language Models during Decodingcited by 8
  22. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  23. 2024A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Modelscited by 7
  24. 2025MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Trainingcited by 7
  25. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  26. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  27. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  28. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Expertscited by 7
  29. 2024Performance Law of Large Language Modelscited by 6
  30. 2024First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Modelscited by 6
  31. 2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
  32. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  33. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  34. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Modelscited by 6
  35. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
  36. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  37. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  38. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  2. 2025Mixture of Experts in Large Language Models
  3. 2024A Survey on Mixture of Experts in Large Language Models
  4. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  5. 2024Scaling Laws for Fine-Grained Mixture of Experts
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  7. 2025dots.llm1 Technical Report
  8. 2021Efficient Large Scale Language Modeling with Mixtures of Experts