# Mutual Adversarial Training: Learning together is better than going alone

arXiv: https://arxiv.org/abs/2112.05005 · HTML: https://mlsanity.com/paper/2112.05005/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [Adversarial Training and Robustness for Multiple Perturbations](https://mlsanity.com/paper/1904.13000/) (2019)
- [The Space of Transferable Adversarial Examples](https://mlsanity.com/paper/1704.03453/) (2017)
- [EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples](https://mlsanity.com/paper/1709.04114/) (2017)
- [Perceptual Adversarial Robustness: Defense Against Unseen Threat Models](https://mlsanity.com/paper/2006.12655/) (2020)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defenses](https://mlsanity.com/paper/1811.09600/) (2018)
- [Towards the first adversarially robust neural network model on MNIST](https://mlsanity.com/paper/1805.09190/) (2018)
- [(De)Randomized Smoothing for Certifiable Defense against Patch Attacks](https://mlsanity.com/paper/2002.10733/) (2020)
- [Adversarial Robustness Against the Union of Multiple Perturbation Models](https://mlsanity.com/paper/1909.04068/) (2019)
- [Functional Adversarial Attacks](https://mlsanity.com/paper/1906.00001/) (2019)

## Built on by (most cited first, 17 of 17)

- [Instruct2Attack: Language-Guided Semantic Adversarial Attacks](https://mlsanity.com/paper/2311.15551/) (2023)
- [A White-Box False Positive Adversarial Attack Method on Contrastive Loss Based Offline Handwritten Signature Verification Models](https://mlsanity.com/paper/2308.08925/) (2023)
- [Defenses in Adversarial Machine Learning: A Survey](https://mlsanity.com/paper/2312.08890/) (2023)
- [Instant Adversarial Purification with Adversarial Consistency Distillation](https://mlsanity.com/paper/2408.17064/) (2024)
- [Interpolated Joint Space Adversarial Training for Robust and Generalizable Defenses](https://mlsanity.com/paper/2112.06323/) (2021)
- [Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training](https://mlsanity.com/paper/2401.14948/) (2024)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Improving Fast Adversarial Training via Self-Knowledge Guidance](https://mlsanity.com/paper/2409.17589/) (2024)
- [Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models](https://mlsanity.com/paper/2410.05951/) (2024)
- [ZeroPur: Succinct Training-Free Adversarial Purification](https://mlsanity.com/paper/2406.03143/) (2024)
- [MMAD-Purify: A Precision-Optimized Framework for Efficient and Scalable Multi-Modal Attacks](https://mlsanity.com/paper/2410.14089/) (2024)
- [Parameter Interpolation Adversarial Training for Robust Image Classification](https://mlsanity.com/paper/2511.00836/) (2025)
- [Releasing Inequality Phenomenon in ℓ∞-norm Adversarial Training via Input Gradient Distillation](https://mlsanity.com/paper/2305.09305/) (2023)
- [A Theoretical Perspective on Subnetwork Contributions to Adversarial Robustness](https://mlsanity.com/paper/2307.03803/) (2023)
- [Enhancing Robust Representation in Adversarial Training: Alignment and Exclusion Criteria](https://mlsanity.com/paper/2310.03358/) (2023)
- [Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective](https://mlsanity.com/paper/2408.03944/) (2024)
- [Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Training](https://mlsanity.com/paper/2604.26496/) (2026)

## Abstract (arXiv)

Recent studies have shown that robustness to adversarial attacks can be transferred across networks. In other words, we can make a weak model more robust with the help of a strong teacher model. We ask if instead of learning from a static teacher, can models "learn together" and "teach each other" to achieve better robustness? In this paper, we study how interactions among models affect robustness via knowledge distillation. We propose mutual adversarial training (MAT), in which multiple models are trained together and share the knowledge of adversarial examples to achieve improved robustness. MAT allows robust models to explore a larger space of adversarial samples, and find more robust feature spaces and decision boundaries. Through extensive experiments on CIFAR-10 and CIFAR-100, we demonstrate that MAT can effectively improve model robustness and outperform state-of-the-art methods under white-box attacks, bringing ~8% accuracy gain to vanilla adversarial training (AT) under PGD-100 attacks. In addition, we show that MAT can also mitigate the robustness trade-off among different perturbation types, bringing as much as 13.1% accuracy gain to AT baselines against the union of l_∞, l2 and l1 attacks. These results show the superiority of the proposed method and demonstrate that collaborative learning is an effective strategy for designing robust models.