# AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Framework

arXiv: https://arxiv.org/abs/2310.01818 · HTML: https://mlsanity.com/paper/2310.01818/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Adversarial Examples Improve Image Recognition](https://mlsanity.com/paper/1911.09665/) (2019)
- [Enhancing the Transferability of Adversarial Attacks through Variance Tuning](https://mlsanity.com/paper/2103.15571/) (2021)
- [Robust Pre-Training by Adversarial Contrastive Learning](https://mlsanity.com/paper/2010.13337/) (2020)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2306.07967/) (2023)
- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Adversarially robust transfer learning](https://mlsanity.com/paper/1905.08232/) (2019)
- [Reliable Adversarial Distillation with Unreliable Teachers](https://mlsanity.com/paper/2106.04928/) (2021)
- [Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free](https://mlsanity.com/paper/2010.11828/) (2020)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Adaptive Image Transformations for Transfer-based Adversarial Attack](https://mlsanity.com/paper/2111.13844/) (2021)
- [Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness](https://mlsanity.com/paper/2207.10899/) (2022)
- [TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization](https://mlsanity.com/paper/2303.11135/) (2023)

## Built on by (most cited first, 9 of 9)

- [FouRA: Fourier Low Rank Adaptation](https://mlsanity.com/paper/2406.08798/) (2024)
- [Initialization Matters for Adversarial Transfer Learning](https://mlsanity.com/paper/2312.05716/) (2023)
- [FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers](https://mlsanity.com/paper/2401.01752/) (2024)
- [Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2505.15130/) (2025)
- [Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off](https://mlsanity.com/paper/2402.14648/) (2024)
- [IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring](https://mlsanity.com/paper/2603.13792/) (2026)
- [Transfer Learning for Finetuning Large Language Models](https://mlsanity.com/paper/2411.01195/) (2024)
- [Aggregate, Decompose, and Fine-Tune: A Simple Yet Effective Factor-Tuning Method for Vision Transformer](https://mlsanity.com/paper/2311.06749/) (2023)
- [A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy](https://mlsanity.com/paper/2508.14079/) (2025)

## Abstract (arXiv)

Robust Fine-Tuning (RFT) is a low-cost strategy to obtain adversarial robustness in downstream applications, without requiring a lot of computational resources and collecting significant amounts of data. This paper uncovers an issue with the existing RFT, where optimizing both adversarial and natural objectives through the feature extractor (FE) yields significantly divergent gradient directions. This divergence introduces instability in the optimization process, thereby hindering the attainment of adversarial robustness and rendering RFT highly sensitive to hyperparameters. To mitigate this issue, we propose a low-rank (LoRa) branch that disentangles RFT into two distinct components: optimizing natural objectives via the LoRa branch and adversarial objectives via the FE. Besides, we introduce heuristic strategies for automating the scheduling of the learning rate and the scalars of loss terms. Extensive empirical evaluations demonstrate that our proposed automated RFT disentangled via the LoRa branch (AutoLoRa) achieves new state-of-the-art results across a range of downstream tasks. AutoLoRa holds significant practical utility, as it automatically converts a pre-trained FE into an adversarially robust model for downstream tasks without the need for searching hyperparameters.