# Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

arXiv: https://arxiv.org/abs/2606.18024 · HTML: https://mlsanity.com/paper/2606.18024/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Functional Regularisation for Continual Learning with Gaussian Processes](https://mlsanity.com/paper/1901.11356/) (2019)
- [A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix](https://mlsanity.com/paper/2010.04003/) (2020)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent](https://mlsanity.com/paper/2006.11942/) (2020)
- [Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2601.18699/) (2026)

## Built on by (most cited first, 2 of 2)

- [Omega-S: A Functional Resilience Index for LLM Fine-Tuning](https://mlsanity.com/paper/2608.03887/) (2026)
- [A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks](https://mlsanity.com/paper/2610.09835/) (2026)

## Abstract (arXiv)

Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision; for nonlinear adapters/full fine-tuning, it is a local NTK approximation. The same expression reveals that forgetting concentrates in a small number of old-task NTK eigenmodes and under frozen linear heads gives a Kronecker scaling rule for the vulnerable rank. These results clarify the relation to prior NTK-overlap theory, explain why parameter-space regularizers can miss output-space interference, and motivate a targeted spectral regularizer.