# XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection

arXiv: https://arxiv.org/abs/2403.18926 · HTML: https://mlsanity.com/paper/2403.18926/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [Large Memory Layers with Product Keys](https://mlsanity.com/paper/1907.05242/) (2019)
- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [Sparse is Enough in Scaling Transformers](https://mlsanity.com/paper/2111.12763/) (2021)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [M6-T: Exploring Sparse Expert Models and Beyond](https://mlsanity.com/paper/2105.15082/) (2021)

## Built on by (most cited first, 28 of 28)

- [Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models](https://mlsanity.com/paper/2405.14297/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers](https://mlsanity.com/paper/2503.14487/) (2025)
- [TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding](https://mlsanity.com/paper/2508.01699/) (2025)
- [Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis](https://mlsanity.com/paper/2502.04416/) (2025)
- [Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization](https://mlsanity.com/paper/2509.26520/) (2025)
- [UMoE: Unifying Attention and FFN with Shared Experts](https://mlsanity.com/paper/2505.07260/) (2025)
- [CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts](https://mlsanity.com/paper/2410.16077/) (2024)
- [Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference](https://mlsanity.com/paper/2604.08133/) (2026)
- [DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training](https://mlsanity.com/paper/2512.13996/) (2025)
- [Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing](https://mlsanity.com/paper/2603.11535/) (2026)
- [Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning](https://mlsanity.com/paper/2502.03884/) (2025)
- [SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment](https://mlsanity.com/paper/2606.25821/) (2026)
- [Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations](https://mlsanity.com/paper/2607.20426/) (2026)
- [Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer](https://mlsanity.com/paper/2503.02495/) (2025)
- [BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE](https://mlsanity.com/paper/2605.14438/) (2026)
- [Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference](https://mlsanity.com/paper/2606.29982/) (2026)
- [LoGo: Token-Level Dynamic Local-Global Attention](https://mlsanity.com/paper/2608.29539/) (2026)
- [Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts](https://mlsanity.com/paper/2502.12928/) (2025)
- [FT-MoE: Sustainable-learning Mixture of Experts for Fault-Tolerant Computing](https://mlsanity.com/paper/2504.20446/) (2025)
- [An item is worth one token in Multimodal Large Language Models-based Sequential Recommendation](https://mlsanity.com/paper/2511.05885/) (2025)
- [AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert](https://mlsanity.com/paper/2511.18314/) (2025)
- [PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2512.16494/) (2025)
- [Mtextsuperscript4Fuse: Lightweight State-Space MoE with a Cross-Scale Gating Bridge for Brain Tumor Segmentation](https://mlsanity.com/paper/2605.02444/) (2026)
- [TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts](https://mlsanity.com/paper/2606.09885/) (2026)
- [Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers](https://mlsanity.com/paper/2609.22199/) (2026)
- [EAT: Expert Account Tracker for Efficient MoE Inference](https://mlsanity.com/paper/2609.33614/) (2026)
- [How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts](https://mlsanity.com/paper/2610.19033/) (2026)

## Abstract (arXiv)

Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational inefficiency since a significant number of parameters are unnecessarily involved in computations via multiplying values by zero or low activation values. To address this issue, we present tool, a novel MoE designed to enhance both the efficacy and efficiency of sparse MoE models. tool leverages small experts and a threshold-based router to enable tokens to selectively engage only essential parameters. Our extensive experiments on language modeling and machine translation tasks demonstrate that tool can enhance model performance while decreasing the computation load at MoE layers by over 50% without sacrificing performance. Furthermore, we present the versatility of tool by applying it to dense models, enabling sparse computation during inference. We provide a comprehensive analysis and make our code available at https://github.com/ysngki/XMoE.