# A Dynamical Theory of LoRA in Continual Learning

arXiv: https://arxiv.org/abs/2609.39367 · HTML: https://mlsanity.com/paper/2609.39367/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation](https://mlsanity.com/paper/2411.05663/) (2024)
- [LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)](https://mlsanity.com/paper/2502.09376/) (2025)
- [Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation](https://mlsanity.com/paper/2205.09029/) (2022)
- [Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics](https://mlsanity.com/paper/2505.23194/) (2025)
- [Optimal Protocols for Continual Learning via Statistical Physics and Control Theory](https://mlsanity.com/paper/2409.18061/) (2024)
- [Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization](https://mlsanity.com/paper/2503.06982/) (2025)
- [LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning](https://mlsanity.com/paper/2508.06202/) (2025)
- [Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters](https://mlsanity.com/paper/2506.14530/) (2025)
- [Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models](https://mlsanity.com/paper/2410.16801/) (2024)
- [A Theory of Initialisation's Impact on Specialisation](https://mlsanity.com/paper/2503.02526/) (2025)
- [High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model](https://mlsanity.com/paper/2606.05899/) (2026)
- [When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models](https://mlsanity.com/paper/2602.02855/) (2026)

## Abstract (arXiv)

Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-dimensional online-learning limit, we derive a closed system of ordinary differential equations for a finite set of macroscopic order parameters, yielding exact expressions for the generalization errors throughout both the initial Task 1 learning phase and the subsequent LoRA fine-tuning on Task 2. The theory quantitatively matches finite-dimensional simulations and exposes two characteristic effects of LoRA: low-rank adaptation reduces interference with features learned on the first task, but its initialization slows adaptation to the second task. Building on this mechanistic picture, we analyze a state-dependent masking strategy that freezes hidden units carrying the strongest first-task representations and restricts adaptation to the complementary subspace. This structural partitioning markedly reduces forgetting, while preserving plasticity on the new task. Our framework further clarifies the role of adapter rank: transfer improves only up to the intrinsic dimensionality of the target task and saturates beyond it, while forgetting continues to grow with rank. These results provide a dynamical and geometric account of how low-rank adaptation organizes information across sequential tasks and are qualitatively reproduced on a sequential MNIST benchmark.