# Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

arXiv: https://arxiv.org/abs/2509.05086 · HTML: https://mlsanity.com/paper/2509.05086/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Sparse DNNs with Improved Adversarial Robustness](https://mlsanity.com/paper/1810.09619/) (2018)
- [ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing](https://mlsanity.com/paper/2412.14711/) (2024)
- [Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks](https://mlsanity.com/paper/2306.04073/) (2023)
- [Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments](https://mlsanity.com/paper/1707.05733/) (2017)
- [Robust Mixture-of-Expert Training for Convolutional Neural Networks](https://mlsanity.com/paper/2308.10110/) (2023)
- [Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks](https://mlsanity.com/paper/2110.14068/) (2021)
- [On the Adversarial Robustness of Mixture of Experts](https://mlsanity.com/paper/2210.10253/) (2022)
- [HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts](https://mlsanity.com/paper/2312.07035/) (2023)
- [LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes](https://mlsanity.com/paper/2501.04004/) (2025)
- [Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?](https://mlsanity.com/paper/2401.13875/) (2024)
- [Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability](https://mlsanity.com/paper/2204.10598/) (2022)
- [textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Experts](https://mlsanity.com/paper/2406.11353/) (2024)

## Built on by (most cited first, 4 of 4)

- [Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation](https://mlsanity.com/paper/2604.13761/) (2026)
- [A Comparative Study of Adversarial Robustness in CNN and CNN-ANFIS Architectures](https://mlsanity.com/paper/2602.07028/) (2026)
- [Domain-Specialized Object Detection via Model-Level Mixtures of Experts](https://mlsanity.com/paper/2604.18256/) (2026)
- [RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations](https://mlsanity.com/paper/2607.06109/) (2026)

## Abstract (arXiv)

Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.