# Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation

arXiv: https://arxiv.org/abs/2306.16170 · HTML: https://mlsanity.com/paper/2306.16170/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 62 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack](https://mlsanity.com/paper/1907.02044/) (2019)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Geometry-aware Instance-reweighted Adversarial Training](https://mlsanity.com/paper/2010.01736/) (2020)
- [Disentangling Adversarial Robustness and Generalization](https://mlsanity.com/paper/1812.00740/) (2018)
- [Are Labels Required for Improving Adversarial Robustness?](https://mlsanity.com/paper/1905.13725/) (2019)
- [Enhancing Adversarial Example Transferability with an Intermediate Level Attack](https://mlsanity.com/paper/1907.10823/) (2019)
- [Adversarially Robust Distillation](https://mlsanity.com/paper/1905.09747/) (2019)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)

## Built on by (most cited first, 13 of 13)

- [Towards Viewpoint-Invariant Visual Recognition via Adversarial Training](https://mlsanity.com/paper/2307.10235/) (2023)
- [MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Models](https://mlsanity.com/paper/2511.17448/) (2025)
- [Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing](https://mlsanity.com/paper/2510.01243/) (2025)
- [Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation](https://mlsanity.com/paper/2312.05508/) (2023)
- [TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration](https://mlsanity.com/paper/2410.12183/) (2024)
- [D2R: dual regularization loss with collaborative adversarial generation for model robustness](https://mlsanity.com/paper/2506.07056/) (2025)
- [CIARD: Cyclic Iterative Adversarial Robustness Distillation](https://mlsanity.com/paper/2509.12633/) (2025)
- [Knowledge-Guided Adversarial Training for Infrared Object Detection via Thermal Radiation Modeling](https://mlsanity.com/paper/2603.25170/) (2026)
- [SGD Jittering: A Training Strategy for Robust and Accurate Model-Based Architectures](https://mlsanity.com/paper/2410.14667/) (2024)
- [Feature Statistics with Uncertainty Help Adversarial Robustness](https://mlsanity.com/paper/2503.20583/) (2025)
- [RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selection](https://mlsanity.com/paper/2505.12861/) (2025)
- [Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation](https://mlsanity.com/paper/2506.08611/) (2025)
- [Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers](https://mlsanity.com/paper/2607.27737/) (2026)

## Abstract (arXiv)

Adversarial Training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. Although bringing reliable robustness, the performance towards clean examples is negatively affected after Adversarial Training, which means a trade-off exists between accuracy and robustness. Recently, some studies have tried to use knowledge distillation methods in Adversarial Training, achieving competitive performance in improving the robustness but the accuracy for clean samples is still limited. In this paper, to mitigate the accuracy-robustness trade-off, we introduce the Balanced Multi-Teacher Adversarial Robustness Distillation (B-MTARD) to guide the model's Adversarial Training process by applying a strong clean teacher and a strong robust teacher to handle the clean examples and adversarial examples, respectively. During the optimization process, to ensure that different teachers show similar knowledge scales, we design the Entropy-Based Balance algorithm to adjust the teacher's temperature and keep the teachers' information entropy consistent. Besides, to ensure that the student has a relatively consistent learning speed from multiple teachers, we propose the Normalization Loss Balance algorithm to adjust the learning weights of different types of knowledge. A series of experiments conducted on three public datasets demonstrate that B-MTARD outperforms the state-of-the-art methods against various adversarial attacks.