# Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models

arXiv: https://arxiv.org/abs/2601.06305 · HTML: https://mlsanity.com/paper/2601.06305/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [ONION: A Simple and Effective Defense Against Textual Backdoor Attacks](https://mlsanity.com/paper/2011.10369/) (2020)
- [A backdoor attack against LSTM-based text classification systems](https://mlsanity.com/paper/1905.12457/) (2019)
- [Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger](https://mlsanity.com/paper/2105.12400/) (2021)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identification](https://mlsanity.com/paper/2007.12070/) (2020)
- [Towards Robustness Against Natural Language Word Substitutions](https://mlsanity.com/paper/2107.13541/) (2021)
- [Triggerless Backdoor Attack for NLP Tasks with Clean Labels](https://mlsanity.com/paper/2111.07970/) (2021)
- [BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models](https://mlsanity.com/paper/2406.17092/) (2024)
- [Exploring the Universal Vulnerability of Prompt-based Learning Paradigm](https://mlsanity.com/paper/2204.05239/) (2022)
- [How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?](https://mlsanity.com/paper/2112.11668/) (2021)
- [BadHash: Invisible Backdoor Attacks against Deep Hashing with Clean Label](https://mlsanity.com/paper/2207.00278/) (2022)
- [Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation](https://mlsanity.com/paper/2410.14425/) (2024)
- [LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)](https://mlsanity.com/paper/2502.09376/) (2025)
- [BFClass: A Backdoor-free Text Classification Framework](https://mlsanity.com/paper/2109.10855/) (2021)
- [Textual Manifold-based Defense Against Natural Language Adversarial Examples](https://mlsanity.com/paper/2211.02878/) (2022)
- [Backdoor Attacks with Input-unique Triggers in NLP](https://mlsanity.com/paper/2303.14325/) (2023)
- [Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Space](https://mlsanity.com/paper/2402.12026/) (2024)
- [Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMs](https://mlsanity.com/paper/2409.17946/) (2024)

## Built on by (most cited first, 4 of 4)

- [LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents](https://mlsanity.com/paper/2602.01053/) (2026)
- [High-Dimensional Statistics: Reflections on Progress and Open Problems](https://mlsanity.com/paper/2605.05076/) (2026)
- [Weight space Detection of Backdoors in LoRA Adapters](https://mlsanity.com/paper/2602.15195/) (2026)
- [AGENTQ: Quantization-Conditioned Backdoor Attacks on LLM Agents](https://mlsanity.com/paper/2609.14060/) (2026)

## Abstract (arXiv)

Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models, but it is notably ineffective at removing backdoor behaviors from poisoned pretrained models when fine-tuning on clean dataset. Contrary to the common belief that this weakness is caused primarily by low rank, we show that LoRA's vulnerability is fundamentally spectral. Our analysis identifies two key factors: LoRA updates (i) possess insufficient spectral strength, with singular values far below those of pretrained weights, and (ii) exhibit unfavorable spectral alignment, weakly matching clean-task directions while retaining overlap with trigger-sensitive subspaces. We further establish a critical scaling threshold beyond which LoRA can theoretically suppress trigger-induced activations, and we show empirically that standard LoRA rarely reaches this regime. We introduce Regularized Low-Rank Adaptation (RoRA), which improves forgetting by increasing spectral strength and correcting alignment through clean-strengthened regularization, trigger-insensitive constraints, and post-training spectral rescaling. Experiments across multiple NLP benchmarks and attack settings show that RoRA substantially reduces attack success rates while maintaining clean accuracy.