# Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

arXiv: https://arxiv.org/abs/2404.05567 · HTML: https://mlsanity.com/paper/2404.05567/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 27 of its 76 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 27 of 27)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models](https://mlsanity.com/paper/2305.14705/) (2023)
- [Mixture of Attention Heads: Selecting Attention Heads Per Token](https://mlsanity.com/paper/2210.05144/) (2022)
- [ModuleFormer: Modularity Emerges from Mixture-of-Experts](https://mlsanity.com/paper/2306.04640/) (2023)
- [Unlocking Emergent Modularity in Large Language Models](https://mlsanity.com/paper/2310.10908/) (2023)
- [VA-RED2: Video Adaptive Redundancy Reduction](https://mlsanity.com/paper/2102.07887/) (2021)
- [Scattered Mixture-of-Experts Implementation](https://mlsanity.com/paper/2403.08245/) (2024)

## Built on by (most cited first, 30 of 42)

- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [JetMoE: Reaching Llama2 Performance with 0.1M Dollars](https://mlsanity.com/paper/2404.07413/) (2024)
- [Understanding the Performance and Estimating the Cost of LLM Fine-Tuning](https://mlsanity.com/paper/2408.04693/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [Advancing Expert Specialization for Better MoE](https://mlsanity.com/paper/2505.22323/) (2025)
- [Mixture of Parrots: Experts improve memorization more than reasoning](https://mlsanity.com/paper/2410.19034/) (2024)
- [LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing](https://mlsanity.com/paper/2507.00029/) (2025)
- [Monet: Mixture of Monosemantic Experts for Transformers](https://mlsanity.com/paper/2412.04139/) (2024)
- [DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism](https://mlsanity.com/paper/2504.00661/) (2025)
- [Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition](https://mlsanity.com/paper/2410.17765/) (2024)
- [First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models](https://mlsanity.com/paper/2408.11393/) (2024)
- [Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design](https://mlsanity.com/paper/2504.01337/) (2025)
- [Exploring the Benefit of Activation Sparsity in Pre-training](https://mlsanity.com/paper/2410.03440/) (2024)
- [LaDiMo: Layer-wise Distillation Inspired MoEfier](https://mlsanity.com/paper/2408.04278/) (2024)
- [Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.12476/) (2026)
- [SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs](https://mlsanity.com/paper/2606.17952/) (2026)
- [Dynamic Activation Pitfalls in LLaMA Models: An Empirical Study](https://mlsanity.com/paper/2405.09274/) (2024)
- [TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch](https://mlsanity.com/paper/2412.15622/) (2024)
- [Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging](https://mlsanity.com/paper/2502.01804/) (2025)
- [Sparse Brains are Also Adaptive Brains: Cognitive-Load-Aware Dynamic Activation for LLMs](https://mlsanity.com/paper/2502.19078/) (2025)
- [BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers](https://mlsanity.com/paper/2507.03117/) (2025)
- [Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts](https://mlsanity.com/paper/2601.03577/) (2026)
- [RelMap: Enhancing Online Map Construction with Class-Aware Spatial Relation and Semantic Priors](https://mlsanity.com/paper/2507.21567/) (2025)
- [Routing-Free Mixture-of-Experts](https://mlsanity.com/paper/2604.00801/) (2026)
- [Does a Global Perspective Help Prune Sparse MoEs Elegantly?](https://mlsanity.com/paper/2604.06542/) (2026)
- [Meta-Continual Learning of Neural Fields](https://mlsanity.com/paper/2504.05806/) (2025)
- [Multi-Expert Learning Framework with the State Space Model for Optical and SAR Image Registration](https://mlsanity.com/paper/2507.04397/) (2025)
- [UniPool: A Globally Shared Expert Pool for Mixture-of-Experts](https://mlsanity.com/paper/2605.06665/) (2026)
- [MOYU: A Theoretical Study on Massive Over-activation Yielded Uplifts in LLMs](https://mlsanity.com/paper/2406.12569/) (2024)

## Abstract (arXiv)

Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4× compared to dense models without sacrificing performance, making them more efficient in computation-bounded scenarios. However, MoE models generally require 2-4× times more parameters to achieve comparable performance to a dense model, which incurs larger GPU memory requirements and makes MoE models less efficient in I/O-bounded scenarios like autoregressive generation. In this work, we propose a hybrid dense training and sparse inference framework for MoE models (DS-MoE) which achieves strong computation and parameter efficiency by employing dense computation across all experts during training and sparse computation during inference. Our experiments on training LLMs demonstrate that our DS-MoE models are more parameter-efficient than standard sparse MoEs and are on par with dense models in terms of total parameter size and performance while being computationally cheaper (activating 30-40% of the model's parameters). Performance tests using vLLM show that our DS-MoE-6B model runs up to 1.86× faster than similar dense models like Mistral-7B, and between 1.50× and 1.71× faster than comparable MoEs, such as DeepSeekMoE-16B and Qwen1.5-MoE-A2.7B.