mlsanityData to 7 Oct 2026

MoE large language models

541 papers4 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: moe, language, experts, expert, performance

Key facts. MoE large language models: 541 papers on arXiv since 2017 (data through 7 Oct 2026). 229 of its 541 mapped papers (42%) cite “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” (2017, arXiv 1701.06538). 162 of its 541 mapped papers (30%) cite “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding” (2020, arXiv 2006.16668). 17 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

17 new papers, 17 Sep – 7 Oct; 4 this week.

  1. This week · 4
  2. Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression also fits: Expert routing strategies
  3. Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
  4. VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models
  5. LRCC: Generalizing Low-Rank Compression with Conditional Computation
  6. Earlier · 13
  7. Measuring and Reducing Cross-Vendor Mismatch in Language Models also fits: Efficient MoE training and inference systems
  8. Principled Top-k Selection for Language Models with Hybrid Gradients
  9. ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training also fits: MoE theory and gating functions
  10. Dynamic Expert Pruning for Multi-Agent Systems also fits: Expert routing strategies
  11. Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs
  12. Cross-Lingual Alignment for Decoder-Only Models using MoE Routers also fits: Multimodal MoE models
All 17
  1. ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models
  2. ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control also fits: Expert routing strategies
  3. OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
  4. KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling
  5. Hunyuan-A13B Technical Report
  6. You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs also fits: Expert routing strategies
  7. Accelerating Dense LLMs via L0-regularized Mixture-of-Experts

Foundations

Most cited by papers here

  1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer2017cited by 229 here
  2. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding2020cited by 162 here
  3. Mixtral of Experts2024cited by 110 here
  4. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models2024cited by 102 here
  5. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts2021cited by 76 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 229 here
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 162 here
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 29 here
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 43 here
  5. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 102 here
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 63 here
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 55 here
  8. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 23 here
16 more it builds on, not in this chain
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 214 here
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 40 here
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 76 here
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 64 here
  5. 2022Mixture-of-Experts with Expert Choice Routingcited by 64 here
  6. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 33 here
  7. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 25 here
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 31 here
  9. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 49 here
  10. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 37 here
  11. 2024Mixtral of Expertscited by 110 here
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 39 here
  13. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 46 here
  14. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 23 here
  15. 2024DeepSeek-V3 Technical Reportcited by 45 here
  16. 2025Qwen3 Technical Reportcited by 56 here

The 24 most cited of 489.

Browse

30 of 541, ranked 6–35

  1. 6ST-MoE: Designing Stable and Transferable Sparse Expert Models202264
  2. 7DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model202463
  3. 8Qwen3 Technical Report202556
  4. 9OLMoE: Open Mixture-of-Experts Language Models202455
  5. 10Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints202249
  6. 11Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models202446
  7. 12DeepSeek-V3 Technical Report202445
  8. 13OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models202439
  9. 14Efficient Large Scale Language Modeling with Mixtures of Experts202129
  10. 15Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs202423
All 30
  1. 16MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router202423
  2. 17Scaling Laws for Fine-Grained Mixture of Experts202422
  3. 18LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training202421
  4. 19Scalable and Efficient MoE Training for Multitask Multilingual Models202121
  5. 20Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models202419
  6. 21MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition202419
  7. 22Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference202119
  8. 23Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM202418
  9. 24Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts202417
  10. 25Mixture Compressor for Mixture-of-Experts LLMs Gains More202416
  11. 26Qwen2.5 Technical Report202416
  12. 27Delta Decompression for MoE-based LLMs Compression202515
  13. 28Upcycling Large Language Models into Mixture of Experts202415
  14. 29A Survey on Mixture of Experts in Large Language Models202414
  15. 30Retraining-Free Merging of Sparse MoE via Hierarchical Clustering202414
  16. 31Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models202314
  17. 32MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance202512
  18. 33Jamba: A Hybrid Transformer-Mamba Language Model202412
  19. 34Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques202412
  20. 35MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design202511