mlsanityData to 14 Oct 2026

RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations

arXiv 2607.06109 ↗First posted 7 Jul 2026cs.CV cs.AIcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple ℓp perturbations but suffers from robustness trade-offs between different threats. To address this, we employ a mixture of experts (MoE) to route different threats through distinct model pathways. However, naive application of MoE encounters two critical challenges: experts tend to overlook threat-specific features and redundantly capture features shared across threats, and gating networks suffer from threat-agnostic routing where they learn nearly identical routing patterns across threats, thus preventing the construction of threat-specific model pathways. To this end, we propose Robust Mixture of Low-Rank Experts (RoME), where each expert is a low-rank additive update to the shared backbone, allowing it to capture threat-common features while experts focus on threat-specific information. To address threat-agnostic routing, RoME introduces (i) dual-scale gating that exploits threat-discriminative signals from local and global level features, and (ii) threat-guided gating diversification that enforces diverse expert utilization across threats. Extensive experiments demonstrate that RoME outperforms existing state-of-the-art MAT in union robustness and natural accuracy and improves robustness against unseen threats. Codes are available at https://github.com/wkim97/RoME.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 58 of its 76 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

58 of 76 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers5 Sep
  2. 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach5 Feb
  3. 2024Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation16 Dec
  4. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networks19 Aug
  5. 2023From Sparse to Soft Mixtures of Experts2 Aug
  6. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design26 Oct
  7. 2022On the Adversarial Robustness of Mixture of Experts19 Oct
  8. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 50

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2016Wide Residual Networkscited by 3,070
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  4. 2016Progressive Neural Networkscited by 1,361
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  9. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  10. 2019Adversarial Training for Free!cited by 643
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2017Adversarial Patchcited by 517
  13. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  14. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  15. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  16. 2018Spatially Transformed Adversarial Examplescited by 195
  17. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 183
  18. 2024Mixture of LoRA Expertscited by 181
  19. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  20. 2018SparseFool: a few pixels make a big differencecited by 94
  21. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  22. 2019Adversarial Robustness Against the Union of Multiple Perturbation Modelscited by 88
  23. 2019Functional Adversarial Attackscited by 82
  24. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  25. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  26. 2024Theory on Mixture-of-Experts in Continual Learningcited by 54
  27. 2024Revisiting Adversarial Training at Scalecited by 33
  28. 2018Humans can decipher adversarial imagescited by 31
  29. 2021Adversarial Robustness against Multiple and Single lp-Threat Models via Quick Fine-Tuning of Robust Classifierscited by 23
  30. 2023Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shiftscited by 23
  31. 2020Learning to Generate Noise for Multi-Attack Robustnesscited by 17
  32. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16
  33. 2022Formulating Robustness Against Unforeseen Attackscited by 8
  34. 2024PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learningcited by 8
  35. 2021Mixture of Robust Experts (MoRE):A Robust Denoising Method towards multiple perturbationscited by 6
  36. 2023OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shiftcited by 4
  37. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  38. 2025Adapting to Evolving Adversaries with Regularized Continual Robust Trainingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment
  2. 2021Mutual Adversarial Training: Learning together is better than going alone
  3. 2026RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
  4. 2025Divided We Fall: Defending Against Adversarial Attacks via Soft-Gated Fractional Mixture-of-Experts with Randomized Adversarial Training
  5. 2025Backdoor or Manipulation? Graph Mixture of Experts Can Defend Against Various Graph Adversarial Attacks
  6. 2026Robust CurveMoE: Multi-Norm Adversarial Defense for Mixture-of-Experts Models via Mode Connectivity
  7. 2024Enhancing the "Immunity" of Mixture-of-Experts Networks for Adversarial Defense
  8. 2026Misrouter: Exploiting Routing Mechanisms for Input-Only Attacks on Mixture-of-Experts LLMs