Robustness of Mixtures of Experts to Feature Noise
On the map: Mixture-of-experts → MoE theory and gating functions
Abstract (arXiv)
Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regime where inputs exhibit latent modular structure but are corrupted by feature noise, a proxy for noisy internal activations. We show that sparse expert activation acts as a noise filter: compared to a dense estimator, MoEs achieve lower generalization error under feature noise, improved robustness to perturbations, and faster convergence speed. Empirical results on synthetic data and real-world language tasks corroborate the theoretical insights, demonstrating consistent robustness and efficiency gains from sparse modular computation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 49 of its 63 references that are in the map. First posted on arXiv 21 Jan 2026.
Builds on
49 of 63 references.
Closest ancestors · 7
Same direction, newest first
- 2025On Linear Mode Connectivity of Mixture-of-Experts Architectures14 Sep
- 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks30 May
- 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts22 Oct
- 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep
- 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts12 May
- 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts5 May
- 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul
Other references · 42
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
30 more
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2024Mixtral of Expertscited by 1,823
- 2020GLU Variants Improve Transformercited by 1,804
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2021Scaling Vision with Sparse Mixture of Expertscited by 747
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
- 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
- 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
- 2015Conditional Computation in Neural Networks for faster modelscited by 167
- 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
- 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
- 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
- 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
- 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
- 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
- 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
- 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
- 2024Theory on Mixture-of-Experts in Continual Learningcited by 54
- 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
- 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
- 2022On the Adversarial Robustness of Mixture of Expertscited by 28
- 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 21
- 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 18
- 2020Benign Overfitting and Noisy Featurescited by 18
- 2023Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoiserscited by 14
- 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 12
- 2023SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classificationcited by 10
- 2025The power of fine-grained experts: Granularity boosts expressivity in Mixture of Expertscited by 9
- 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approachcited by 8
- 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
- 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
- 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
- 2025Sparsity and Superposition in Mixture of Experts
- 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
- 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity
- 2021Sparse MoEs meet Efficient Ensembles
- 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts