mlsanityData to 7 Oct 2026

Enhancing the "Immunity" of Mixture-of-Experts Networks for Adversarial Defense

arXiv 2402.18787 ↗First posted 29 Feb 2024cs.LG cs.CRcited by 3

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Recent studies have revealed the vulnerability of Deep Neural Networks (DNNs) to adversarial examples, which can easily fool DNNs into making incorrect predictions. To mitigate this deficiency, we propose a novel adversarial defense method called "Immunity" (Innovative MoE with MUtual information & positioN stabilITY) based on a modified Mixture-of-Experts (MoE) architecture in this work. The key enhancements to the standard MoE are two-fold: 1) integrating of Random Switch Gates (RSGs) to obtain diverse network structures via random permutation of RSG parameters at evaluation time, despite of RSGs being determined after one-time training; 2) devising innovative Mutual Information (MI)-based and Position Stability-based loss functions by capitalizing on Grad-CAM's explanatory power to increase the diversity and the causality of expert networks. Notably, our MI-based loss operates directly on the heatmaps, thereby inducing subtler negative impacts on the classification performance when compared to other losses of the same type, theoretically. Extensive evaluation validates the efficacy of the proposed approach in improving adversarial robustness against a wide range of attacks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 36 of its 45 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

36 of 45 references.

References · 36

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  6. 2016Adversarial examples in the physical worldcited by 1,762
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2016Adversarial Machine Learning at Scalecited by 934
  9. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  10. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  11. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  12. 2018MINE: Mutual Information Neural Estimationcited by 680
24 more
  1. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
  2. 2017Countering Adversarial Images using Input Transformationscited by 517
  3. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  4. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  5. 2019Adversarial Examples Improve Image Recognitioncited by 283
  6. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  7. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  8. 2018Adversarial Logit Pairingcited by 224
  9. 2017Towards Robust Neural Networks via Random Self-ensemblecited by 158
  10. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  11. 2017The Consciousness Priorcited by 105
  12. 2018ComDefend: An Efficient Image Compression Model to Defend Adversarial Examplescited by 104
  13. 2019When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attackscited by 87
  14. 2018Deflecting Adversarial Attacks with Pixel Deflectioncited by 71
  15. 2019Learning Disentangled Representations via Mutual Information Estimationcited by 52
  16. 2020SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localizationcited by 46
  17. 2022Practical Evaluation of Adversarial Robustness via Adaptive Auto Attackcited by 44
  18. 2017Learning Independent Features with Adversarial Nets for Non-linear ICAcited by 33
  19. 2021AdvRush: Searching for Adversarially Robust Neural Architecturescited by 28
  20. 2022Improving Adversarial Robustness via Mutual Information Estimationcited by 13
  21. 2020Improving Interpretability in Medical Imaging Diagnosis using Adversarial Trainingcited by 9
  22. 2019Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defensescited by 8
  23. 2020Provably Robust Adversarial Examplescited by 8
  24. 2019AuxBlocks: Defense Adversarial Example via Auxiliary Blockscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentationcited by 5
  2. 2026Exposing and Defending the Achilles' Heel of Video Mixture-of-Expertscited by 3
  3. 2025Divided We Fall: Defending Against Adversarial Attacks via Soft-Gated Fractional Mixture-of-Experts with Randomized Adversarial Trainingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers
  2. 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
  3. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment
  4. 2025Backdoor or Manipulation? Graph Mixture of Experts Can Defend Against Various Graph Adversarial Attacks
  5. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networks
  6. 2022Towards Understanding Mixture of Experts in Deep Learning
  7. 2026RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations
  8. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs