mlsanityData to 14 Oct 2026

Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

arXiv 2509.05086 ↗First posted 5 Sep 2025cs.CV cs.LGcited by 4

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 26 of its 34 references that are in the map. First posted on arXiv 5 Sep 2025.

Builds on

26 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes7 Jan
  2. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networks19 Aug
  3. 2022On the Adversarial Robustness of Mixture of Experts19 Oct
  4. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability22 Apr
  5. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
9 more
  1. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  2. 2018Sparse DNNs with Improved Adversarial Robustnesscited by 72
  3. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  4. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  5. 2017Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environmentscited by 34
  6. 2021Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networkscited by 29
  7. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
  8. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  9. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Expertscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentationcited by 1
  2. 2026A Comparative Study of Adversarial Robustness in CNN and CNN-ANFIS Architecturescited by 0
  3. 2026Domain-Specialized Object Detection via Model-Level Mixtures of Expertscited by 0
  4. 2026RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
  2. 2024Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation
  3. 2022Towards Understanding Mixture of Experts in Deep Learning
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2018Deep Mixture of Experts via Shallow Embedding
  6. 2024Enhancing the "Immunity" of Mixture-of-Experts Networks for Adversarial Defense
  7. 2026Robustness of Mixtures of Experts to Feature Noise
  8. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment