mlsanityData to 7 Oct 2026

Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics

arXiv 2402.15415 ↗First posted 23 Feb 2024cs.LG math.DS stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. We study forgetting through a tractable _mean-field self-attention_ toy model, where tokens evolve as an interacting particle system and LoRA acts as a low-rank perturbation. Using tools from partial differential equations and dynamical systems, we characterize regimes suggesting a phase transition between forgetting and non-forgetting behavior. We show that one phase transition appears with respect to the norm of the perturbation, and the other with respect to the depth of the Transformers. We further bound the time-to-deviation in terms of the perturbation size and spectral quantities, and corroborate the predicted trends with experiments and exploratory analyses on real models under LoRA fine-tuning.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 23 of its 62 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

23 of 62 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023Mistral 7Bcited by 3,065
  6. 2016Learning without Forgettingcited by 2,465
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  8. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  9. 2017Stable Architectures for Deep Neural Networkscited by 340
  10. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  11. 2023A mathematical perspective on Transformerscited by 160
  12. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
11 more
  1. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  2. 2023The emergence of clusters in self-attention dynamicscited by 133
  3. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  4. 2022Rank Diminishing in Deep Neural Networkscited by 47
  5. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  6. 2020A Mathematical Theory of Attentioncited by 43
  7. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  8. 2020A Functional Perspective on Learning Symmetric Functions with Neural Networkscited by 16
  9. 2023Are More Layers Beneficial to Graph Transformers?cited by 14
  10. 2018Stochastic Deep Networkscited by 13
  11. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024Dynamic metastability in the self-attention modelcited by 42
  3. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  4. 2025Synchronization of mean-field models on the circlecited by 10
  5. 2025Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?cited by 4
  6. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  7. 2026The Intruder Threshold: A Spectral Law for LoRA Fine-Tuningcited by 1