mlsanityData to 13 Oct 2026

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

arXiv 2401.06066 ↗First posted 11 Jan 2024cs.CLcited by 853

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-K out of N experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into mN ones and activating mK from them, allowing for a more flexible combination of activated experts; (2) isolating Ks experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.

Key facts. Cited by 853 papers in the map (data through 13 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 49 of its 65 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

49 of 65 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation15 Dec
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 44

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  12. 2021Program Synthesis with Large Language Modelscited by 3,573
32 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  13. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,289
  14. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 890
  15. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 854
  16. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
  17. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 696
  18. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 634
  19. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 626
  20. 2022Mixture-of-Experts with Expert Choice Routingcited by 619
  21. 2021Knowledge Neurons in Pretrained Transformerscited by 597
  22. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 439
  23. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 402
  24. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 301
  25. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  26. 2021Hash Layers For Large Sparse Modelscited by 237
  27. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 117
  28. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  29. 2021M6: A Chinese Multimodal Pretrainercited by 91
  30. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  31. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  32. 2023Improving Expert Specialization in Mixture of Expertscited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 144; 2025: 349; 2026: 360.

853 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,510
  2. 2024Qwen2.5 Technical Reportcited by 4,840
  3. 2024Qwen2 Technical Reportcited by 2,102
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,450
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,115
  6. 2024DeepSeek-V3 Technical Reportcited by 777
  7. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 566
  8. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 398
  9. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 298
  11. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 286
  12. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 270
38 more
  1. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 265
  2. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 251
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 235
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  5. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 200
  6. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 190
  7. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 186
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 177
  9. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 156
  10. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  11. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 126
  12. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 123
  13. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  14. 2026Kimi K3: Open Frontier Intelligencecited by 122
  15. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 119
  16. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 113
  17. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  18. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 107
  19. 2025UMA: A Family of Universal Models for Atomscited by 102
  20. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 92
  21. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  22. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 89
  23. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  24. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  25. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 81
  26. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  27. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 72
  28. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  29. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 66
  30. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  31. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  32. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 62
  33. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 59
  34. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  35. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  36. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 56
  37. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  38. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55

50 most cited of 853 All 853 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating
  2. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  3. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
  4. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
  5. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  6. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  7. 2024Scaling Laws for Fine-Grained Mixture of Experts
  8. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns