mlsanityData to 7 Oct 2026

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

arXiv 2406.06563 ↗First posted 3 Jun 2024cs.CL cs.AIcited by 57

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In this technical report, we introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is initialized from the pre-existing dense checkpoints of our Skywork-13B model. We explore the comparative effectiveness of upcycling versus training from scratch initializations. Our findings suggest that the choice between these two approaches should consider both the performance of the existing dense checkpoints and the MoE training budget. We highlight two innovative techniques: gating logit normalization, which improves expert diversification, and adaptive auxiliary loss coefficients, allowing for layer-specific adjustment of auxiliary loss coefficients. Our experimental results validate the effectiveness of these methods. Leveraging these techniques and insights, we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile corpus. The evaluation results demonstrate that our model delivers strong performance across a wide range of benchmarks.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 24 of its 28 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

24 of 28 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  2. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
6 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  3. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  6. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 28; 2026: 12.

57 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  3. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  4. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  5. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  6. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  7. 2025Scaling Laws for Native Multimodal Modelscited by 47
  8. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  9. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  10. 2025Advancing Expert Specialization for Better MoEcited by 37
  11. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  12. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
38 more
  1. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  2. 2025Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initializationcited by 23
  3. 2024GRIN: GRadient-INformed MoEcited by 16
  4. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  5. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 12
  6. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  7. 2025Understanding Stragglers in Large Model Training Using What-if Analysiscited by 12
  8. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  9. 2025How to Teach Large Multimodal Models New Skillscited by 10
  10. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 8
  11. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  12. 2025DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Expertscited by 7
  13. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resourcecited by 7
  14. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  15. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  16. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
  17. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  18. 2024MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Trainingcited by 4
  19. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learningcited by 4
  20. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4
  21. 2025Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentationcited by 3
  22. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Expertscited by 3
  23. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Expertscited by 3
  24. 2025GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Modelscited by 2
  25. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 2
  26. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applicationscited by 2
  27. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  28. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 2
  29. 2024Animating the Past: Reconstruct Trilobite via Video Generationcited by 1
  30. 2024Sparse Upcycling: Inference Inefficient Finetuningcited by 1
  31. 2025Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformercited by 1
  32. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  33. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  34. 2025MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Expertscited by 0
  35. 2025Stochastic activationscited by 0
  36. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 0
  37. 2025RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailingcited by 0
  38. 2026Universal Pansharpening Modelcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  3. 2026Training-Free Dynamic Upcycling of Expert Language Models
  4. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  5. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  6. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  8. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models