# On the Adversarial Robustness of Mixture of Experts

arXiv: https://arxiv.org/abs/2210.10253 · HTML: https://mlsanity.com/paper/2210.10253/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Are Labels Required for Improving Adversarial Robustness?](https://mlsanity.com/paper/1905.13725/) (2019)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Go Wider Instead of Deeper](https://mlsanity.com/paper/2107.11817/) (2021)
- [Sparse MoEs meet Efficient Ensembles](https://mlsanity.com/paper/2110.03360/) (2021)

## Built on by (most cited first, 28 of 28)

- [Robust Mixture-of-Expert Training for Convolutional Neural Networks](https://mlsanity.com/paper/2308.10110/) (2023)
- [Routers in Vision Mixture of Experts: An Empirical Study](https://mlsanity.com/paper/2401.15969/) (2024)
- [A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts](https://mlsanity.com/paper/2405.16646/) (2024)
- [GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts](https://mlsanity.com/paper/2312.04693/) (2023)
- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?](https://mlsanity.com/paper/2411.18797/) (2024)
- [WDMoE: Wireless Distributed Mixture of Experts for Large Language Models](https://mlsanity.com/paper/2411.06681/) (2024)
- [MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts](https://mlsanity.com/paper/2410.14574/) (2024)
- [Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach](https://mlsanity.com/paper/2502.06832/) (2025)
- [Tight Clusters Make Specialized Experts](https://mlsanity.com/paper/2502.15315/) (2025)
- [A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness](https://mlsanity.com/paper/2309.03004/) (2023)
- [Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation](https://mlsanity.com/paper/2412.11608/) (2024)
- [Dense Hopfield Networks in the Teacher-Student Setting](https://mlsanity.com/paper/2401.04191/) (2024)
- [Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learning](https://mlsanity.com/paper/2405.15984/) (2024)
- [Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers](https://mlsanity.com/paper/2509.05086/) (2025)
- [Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures](https://mlsanity.com/paper/2510.16411/) (2025)
- [Certifying Ensembles: A General Certification Theory with S-Lipschitzness](https://mlsanity.com/paper/2304.13019/) (2023)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Divided We Fall: Defending Against Adversarial Attacks via Soft-Gated Fractional Mixture-of-Experts with Randomized Adversarial Training](https://mlsanity.com/paper/2512.20821/) (2025)
- [Robustness of Mixtures of Experts to Feature Noise](https://mlsanity.com/paper/2601.14792/) (2026)
- [A Wolf in Sheep's Clothing: Targeted Routing Hijacking in Federated RAG](https://mlsanity.com/paper/2605.28112/) (2026)
- [Backdoor or Manipulation? Graph Mixture of Experts Can Defend Against Various Graph Adversarial Attacks](https://mlsanity.com/paper/2510.15333/) (2025)
- [Teacher-Guided Routing for Sparse Vision Mixture-of-Experts](https://mlsanity.com/paper/2604.21330/) (2026)
- [Towards the Connection between Activation Sparsity and Flat Minima](https://mlsanity.com/paper/2605.25612/) (2026)
- [Explaining Machine Learning and Memorization with Statistical Mechanics](https://mlsanity.com/paper/2606.31110/) (2026)
- [RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations](https://mlsanity.com/paper/2607.06109/) (2026)
- [Robust CurveMoE: Multi-Norm Adversarial Defense for Mixture-of-Experts Models via Mode Connectivity](https://mlsanity.com/paper/2608.26043/) (2026)
- [SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment](https://mlsanity.com/paper/2609.02293/) (2026)

## Abstract (arXiv)

Adversarial robustness is a key desirable property of neural networks. It has been empirically shown to be affected by their sizes, with larger networks being typically more robust. Recently, Bubeck and Sellke proved a lower bound on the Lipschitz constant of functions that fit the training data in terms of their number of parameters. This raises an interesting open question, do -- and can -- functions with more parameters, but not necessarily more computational cost, have better robustness? We study this question for sparse Mixture of Expert models (MoEs), that make it possible to scale up the model size for a roughly constant computational cost. We theoretically show that under certain conditions on the routing and the structure of the data, MoEs can have significantly smaller Lipschitz constants than their dense counterparts. The robustness of MoEs can suffer when the highest weighted experts for an input implement sufficiently different functions. We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost. We make key observations showing the robustness of MoEs to the choice of experts, highlighting the redundancy of experts in models trained in practice.