mlsanityData to 13 Oct 2026

XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection

arXiv 2403.18926 ↗First posted 27 Feb 2024cs.LG cs.CLcited by 27

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational inefficiency since a significant number of parameters are unnecessarily involved in computations via multiplying values by zero or low activation values. To address this issue, we present tool, a novel MoE designed to enhance both the efficacy and efficiency of sparse MoE models. tool leverages small experts and a threshold-based router to enable tokens to selectively engage only essential parameters. Our extensive experiments on language modeling and machine translation tasks demonstrate that tool can enhance model performance while decreasing the computation load at MoE layers by over 50% without sacrificing performance. Furthermore, we present the versatility of tool by applying it to dense models, enabling sparse computation during inference. We provide a comprehensive analysis and make our code available at https://github.com/ysngki/XMoE.

Key facts. Cited by 27 papers in the map (data through 13 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 29 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

23 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 19

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2016Pointer Sentinel Mixture Modelscited by 3,047
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 619
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 402
  11. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  12. 2023From Sparse to Soft Mixtures of Expertscited by 247
7 more
  1. 2021Hash Layers For Large Sparse Modelscited by 237
  2. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 150
  3. 2019Large Memory Layers with Product Keyscited by 121
  4. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  5. 2021Sparse is Enough in Scaling Transformerscited by 86
  6. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  7. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 11.

27 papers.

Sort:

By area:

  1. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2025DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformerscited by 23
  4. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  5. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  6. 2025UMoE: Unifying Attention and FFN with Shared Expertscited by 6
  7. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 6
  8. 2024CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Expertscited by 5
  9. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
  10. 2026Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inferencecited by 3
  11. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuningcited by 2
  12. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 2
15 more
  1. 2026Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinationscited by 2
  2. 2025Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformercited by 1
  3. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  4. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  5. 2026LoGo: Token-Level Dynamic Local-Global Attentioncited by 1
  6. 2025Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Expertscited by 0
  7. 2025FT-MoE: Sustainable-learning Mixture of Experts for Fault-Tolerant Computingcited by 0
  8. 2025An item is worth one token in Multimodal Large Language Models-based Sequential Recommendationcited by 0
  9. 2025AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expertcited by 0
  10. 2025PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimationcited by 0
  11. 2026Mtextsuperscript4Fuse: Lightweight State-Space MoE with a Cross-Scale Gating Bridge for Brain Tumor Segmentationcited by 0
  12. 2026TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Expertscited by 0
  13. 2026SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignmentcited by 0
  14. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformerscited by 0
  15. 2026EAT: Expert Account Tracker for Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  3. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  4. 2025Efficiently Editing Mixture-of-Experts Models with Compressed Experts
  5. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
  6. 2021Taming Sparsely Activated Transformer with Stochastic Experts
  7. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference