# A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning

arXiv: https://arxiv.org/abs/2012.13628 · HTML: https://mlsanity.com/paper/2012.13628/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Dec 2020.

## Built on by (most cited first, 30 of 30)

- [Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models](https://mlsanity.com/paper/2303.01870/) (2023)
- [Current state of LLM Risks and AI Guardrails](https://mlsanity.com/paper/2406.12934/) (2024)
- [Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness](https://mlsanity.com/paper/2110.10942/) (2021)
- [Adversarial Robustness against Multiple and Single lp-Threat Models via Quick Fine-Tuning of Robust Classifiers](https://mlsanity.com/paper/2105.12508/) (2021)
- [TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization](https://mlsanity.com/paper/2303.11135/) (2023)
- [Robust Trajectory Prediction against Adversarial Attacks](https://mlsanity.com/paper/2208.00094/) (2022)
- [On The Empirical Effectiveness of Unrealistic Adversarial Hardening Against Realistic Adversarial Attacks](https://mlsanity.com/paper/2202.03277/) (2022)
- [Improving Robustness by Enhancing Weak Subnets](https://mlsanity.com/paper/2201.12765/) (2022)
- [ASAM: Boosting Segment Anything Model with Adversarial Tuning](https://mlsanity.com/paper/2405.00256/) (2024)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [Adversarial Perturbations Are Not So Weird: Entanglement of Robust and Non-Robust Features in Neural Network Classifiers](https://mlsanity.com/paper/2102.05110/) (2021)
- [Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4](https://mlsanity.com/paper/2203.06649/) (2022)
- [Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoff](https://mlsanity.com/paper/2308.16454/) (2023)
- [Curvature Tuning: Provable Training-free Model Steering From a Single Parameter](https://mlsanity.com/paper/2502.07783/) (2025)
- [On the Robustness Tradeoff in Fine-Tuning](https://mlsanity.com/paper/2503.14836/) (2025)
- [Adversarially Pretrained Transformers May Be Universally Robust In-Context Learners](https://mlsanity.com/paper/2505.14042/) (2025)
- [ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients](https://mlsanity.com/paper/2108.04217/) (2021)
- [Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?](https://mlsanity.com/paper/2108.12275/) (2021)
- [Adversarial Fine-tune with Dynamically Regulated Adversary](https://mlsanity.com/paper/2204.13232/) (2022)
- [Identifying Layers Susceptible to Adversarial Attacks](https://mlsanity.com/paper/2107.04827/) (2021)
- [Turning a Curse into a Blessing: Enabling In-Distribution-Data-Free Backdoor Removal via Stabilized Model Inversion](https://mlsanity.com/paper/2206.07018/) (2022)
- [Guiding the retraining of convolutional neural networks against adversarial inputs](https://mlsanity.com/paper/2207.03689/) (2022)
- [An Empirical Study of the Influence of Adversarial Fine-Tuning on Compressed Neural Networks](https://mlsanity.com/paper/2403.09441/) (2024)
- [Generative AI-based data augmentation for improved bioacoustic classification in noisy environments](https://mlsanity.com/paper/2412.01530/) (2024)
- [A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy](https://mlsanity.com/paper/2508.14079/) (2025)
- [Model X-Ray: Detection of Hidden Malware in AI Model Weights using Few Shot Learning](https://mlsanity.com/paper/2409.19310/) (2024)
- [Efficient Semi-Supervised Adversarial Training via Latent Clustering-Based Data Reduction](https://mlsanity.com/paper/2501.10466/) (2025)
- [What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models](https://mlsanity.com/paper/2603.12799/) (2026)
- [Learning Robustness at Test-Time from a Non-Robust Teacher](https://mlsanity.com/paper/2604.11590/) (2026)

## Abstract (arXiv)

Adversarial Training (AT) with Projected Gradient Descent (PGD) is an effective approach for improving the robustness of the deep neural networks. However, PGD AT has been shown to suffer from two main limitations: i) high computational cost, and ii) extreme overfitting during training that leads to reduction in model generalization. While the effect of factors such as model capacity and scale of training data on adversarial robustness have been extensively studied, little attention has been paid to the effect of a very important parameter in every network optimization on adversarial robustness: the learning rate. In particular, we hypothesize that effective learning rate scheduling during adversarial training can significantly reduce the overfitting issue, to a degree where one does not even need to adversarially train a model from scratch but can instead simply adversarially fine-tune a pre-trained model. Motivated by this hypothesis, we propose a simple yet very effective adversarial fine-tuning approach based on a textitslow start, fast decay learning rate scheduling strategy which not only significantly decreases computational cost required, but also greatly improves the accuracy and robustness of a deep neural network. Experimental results show that the proposed adversarial fine-tuning approach outperforms the state-of-the-art methods on CIFAR-10, CIFAR-100 and ImageNet datasets in both test accuracy and the robustness, while reducing the computational cost by 8-10×. Furthermore, a very important benefit of the proposed adversarial fine-tuning approach is that it enables the ability to improve the robustness of any pre-trained deep neural network without needing to train the model from scratch, which to the best of the authors' knowledge has not been previously demonstrated in research literature.