# MoE large language models

Area: [Mixture-of-experts](https://mlsanity.com/moe/index.md). HTML: https://mlsanity.com/moe/moe-large-language-models/

## Key facts

- MoE large language models: 541 papers on arXiv since 2017 (data through 7 Oct 2026).
- 229 of its 541 mapped papers (42%) cite “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” (2017, arXiv 1701.06538).
- 162 of its 541 mapped papers (30%) cite “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding” (2020, arXiv 2006.16668).
- 17 new papers 17 Sep–7 Oct.

## Most cited here

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017), cited by 229 papers here
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020), cited by 162 papers here
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024), cited by 110 papers here
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024), cited by 102 papers here
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021), cited by 76 papers here
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022), cited by 64 papers here
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024), cited by 63 papers here
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025), cited by 56 papers here
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024), cited by 55 papers here
- [Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://mlsanity.com/paper/2212.05055/) (2022), cited by 49 papers here

## New (17 Sep–7 Oct)

- [Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression](https://mlsanity.com/paper/2610.09342/) (2026-10-07)
- [Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers](https://mlsanity.com/paper/2610.09003/) (2026-10-06)
- [VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models](https://mlsanity.com/paper/2610.07606/) (2026-10-06)
- [LRCC: Generalizing Low-Rank Compression with Conditional Computation](https://mlsanity.com/paper/2610.08858/) (2026-10-05)
- [Measuring and Reducing Cross-Vendor Mismatch in Language Models](https://mlsanity.com/paper/2610.05458/) (2026-10-04)
- [Principled Top-k Selection for Language Models with Hybrid Gradients](https://mlsanity.com/paper/2610.04162/) (2026-10-03)
- [ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training](https://mlsanity.com/paper/2610.04140/) (2026-10-02)
- [Dynamic Expert Pruning for Multi-Agent Systems](https://mlsanity.com/paper/2610.02951/) (2026-10-02)
- [Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs](https://mlsanity.com/paper/2610.02657/) (2026-10-02)
- [Cross-Lingual Alignment for Decoder-Only Models using MoE Routers](https://mlsanity.com/paper/2610.01921/) (2026-10-01)
- [ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models](https://mlsanity.com/paper/2610.01296/) (2026-10-01)
- [ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control](https://mlsanity.com/paper/2609.39137/) (2026-09-30)
- [OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit](https://mlsanity.com/paper/2609.31631/) (2026-09-29)
- [KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling](https://mlsanity.com/paper/2609.27294/) (2026-09-23)
- [Hunyuan-A13B Technical Report](https://mlsanity.com/paper/2609.27284/) (2026-09-23)
- [You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs](https://mlsanity.com/paper/2609.25809/) (2026-09-22)
- [Accelerating Dense LLMs via L0-regularized Mixture-of-Experts](https://mlsanity.com/paper/2609.21672/) (2026-09-18)