mlsanityData to 14 Oct 2026

On the Adversarial Robustness of Mixture of Experts

arXiv 2210.10253 ↗First posted 19 Oct 2022cs.LG cs.AI cs.CRcited by 28

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Adversarial robustness is a key desirable property of neural networks. It has been empirically shown to be affected by their sizes, with larger networks being typically more robust. Recently, Bubeck and Sellke proved a lower bound on the Lipschitz constant of functions that fit the training data in terms of their number of parameters. This raises an interesting open question, do -- and can -- functions with more parameters, but not necessarily more computational cost, have better robustness? We study this question for sparse Mixture of Expert models (MoEs), that make it possible to scale up the model size for a roughly constant computational cost. We theoretically show that under certain conditions on the routing and the structure of the data, MoEs can have significantly smaller Lipschitz constants than their dense counterparts. The robustness of MoEs can suffer when the highest weighted experts for an input implement sufficiently different functions. We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost. We make key observations showing the robustness of MoEs to the choice of experts, highlighting the redundancy of experts in models trained in practice.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 20 of its 32 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

20 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Go Wider Instead of Deeper25 Jul
  2. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 18

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2019On Evaluating Adversarial Robustnesscited by 447
6 more
  1. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  2. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  3. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 278
  4. 2019Are Labels Required for Improving Adversarial Robustness?cited by 163
  5. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  6. 2021Sparse MoEs meet Efficient Ensemblescited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 9; 2025: 6; 2026: 8.

28 papers.

Sort:

By area:

  1. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  2. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 23
  3. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  4. 2023GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Expertscited by 19
  5. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  6. 2024SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?cited by 14
  7. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Modelscited by 12
  8. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  9. 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approachcited by 8
  10. 2025Tight Clusters Make Specialized Expertscited by 8
  11. 2023A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustnesscited by 7
  12. 2024Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentationcited by 5
16 more
  1. 2024Dense Hopfield Networks in the Teacher-Student Settingcited by 4
  2. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  3. 2025Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layerscited by 4
  4. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  5. 2023Certifying Ensembles: A General Certification Theory with S-Lipschitznesscited by 2
  6. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2
  7. 2025Divided We Fall: Defending Against Adversarial Attacks via Soft-Gated Fractional Mixture-of-Experts with Randomized Adversarial Trainingcited by 1
  8. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  9. 2026A Wolf in Sheep's Clothing: Targeted Routing Hijacking in Federated RAGcited by 1
  10. 2025Backdoor or Manipulation? Graph Mixture of Experts Can Defend Against Various Graph Adversarial Attackscited by 0
  11. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Expertscited by 0
  12. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0
  13. 2026Explaining Machine Learning and Memorization with Statistical Mechanicscited by 0
  14. 2026RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbationscited by 0
  15. 2026Robust CurveMoE: Multi-Norm Adversarial Defense for Mixture-of-Experts Models via Mode Connectivitycited by 0
  16. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  2. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  3. 2026Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
  4. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  5. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
  6. 2026Learning Better Certified Models from Empirically-Robust Teachers
  7. 2022Towards Alternative Techniques for Improving Adversarial Robustness: Analysis of Adversarial Training at a Spectrum of Perturbations
  8. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization