# The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning

arXiv: https://arxiv.org/abs/2607.23711 · HTML: https://mlsanity.com/paper/2607.23711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Builds on (most cited first, 29 of 29)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.10512/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning](https://mlsanity.com/paper/2406.09044/) (2024)
- [A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks](https://mlsanity.com/paper/2310.07891/) (2023)
- [Scaling Laws for Forgetting When Fine-Tuning Large Language Models](https://mlsanity.com/paper/2401.05605/) (2024)
- [OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2510.13003/) (2025)
- [Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics](https://mlsanity.com/paper/2402.15415/) (2024)
- [Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario](https://mlsanity.com/paper/2604.18450/) (2026)
- [Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining](https://mlsanity.com/paper/2605.20296/) (2026)
- [High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model](https://mlsanity.com/paper/2606.05899/) (2026)
- [VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuning](https://mlsanity.com/paper/2510.19155/) (2025)
- [Spectral phase transitions and trainability in neural network learning dynamics](https://mlsanity.com/paper/2606.28486/) (2026)
- [GAIN: Multiplicative Modulation for Domain Adaptation](https://mlsanity.com/paper/2604.04516/) (2026)

## Built on by (most cited first, 2 of 2)

- [Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA](https://mlsanity.com/paper/2610.02067/) (2026)
- [How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts](https://mlsanity.com/paper/2610.19033/) (2026)

## Abstract (arXiv)

LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix W+BA that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting. Since their discovery, no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength s^ast=barθ/(γσ1(BA)), computed from the measured spectrum of W alone through the rectangular spiked-deformation transform, together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study spanning four dense Transformer families, a state-space model, a mixture-of-experts model, and an encoder-decoder (18 adapters, 9,840 layer scans), the law localizes the empirical threshold within a factor of two on 82% of layers, separates intruder-bearing from intruder-free layers at deployment with a mean AUC of 0.89, holds unchanged on six third-party adapters, and predicts where WikiText-2 perplexity begins to degrade; a combination of the two pre-specified edge evaluations reaches 98% and is confirmed out-of-bag on the external adapters (0.997). Full fine-tuning disperses its update far below the threshold of every layer, which resolves the asymmetry between LoRA and full fine-tuning. Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting, and a spike-budget rule derived from the thresholds, requiring one SVD and no validation sweeps, reduces forgetting by 62% on the most fragile model at no task cost.