Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability
Not on the map. Why?
Abstract (arXiv)
Catastrophic forgetting in continual learning is often measured at the performance or last-layer representation level, overlooking the underlying mechanisms. We introduce a mechanistic framework that offers a geometric interpretation of catastrophic forgetting as the result of transformations to the encoding of individual features. These transformations can lead to forgetting by reducing the allocated capacity of features or by disrupting their readout by downstream computations. Analysis of a tractable toy model formalizes this view, allowing us to identify best- and worst-case scenarios. Through experiments on this model, we empirically test our formal analysis and highlight the detrimental effect of depth. Finally, we demonstrate how our framework can be used in the analysis of practical models through the use of Crosscoders. We do so through a case study example of a Vision Transformer trained on sequential CIFAR-10. Our work provides a new, feature-centric vocabulary for continual learning.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
39 of 52 references.
References · 39
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2014How transferable are features in deep neural networks?cited by 1,216
- 2022Toy Models of Superpositioncited by 928
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
- 2020Compositional Explanations of Neuronscited by 172
- 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
- 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
27 more
- 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
- 2020A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrixcited by 99
- 2020Optimal Continual Learning has Perfect Memory and is NP-hardcited by 86
- 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
- 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
- 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
- 2022Polysemanticity and Capacity in Neural Networkscited by 74
- 2020Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descentcited by 68
- 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
- 2022Continual evaluation for lifelong learning: Identifying the stability gapcited by 55
- 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
- 2024Mathematical Models of Computation in Superpositioncited by 34
- 2023Continual Learning in Linear Classification on Separable Datacited by 31
- 2022Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regimecited by 28
- 2023Disentangling Neuron Representations with Concept Vectorscited by 28
- 2024Understanding Forgetting in Continual Learning with Linear Regressioncited by 26
- 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
- 2024Revisiting Neural Networks for Continual Learning: An Architectural Perspectivecited by 17
- 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
- 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEscited by 14
- 2024On the Diminishing Returns of Width for Continual Learningcited by 12
- 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 10
- 2024Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Modelscited by 8
- 2025Convergence and Implicit Bias of Gradient Descent on Continual Linear Classificationcited by 8
- 2025Analysis of Overparameterization in Continual Learning under a Linear Modelcited by 4
- 2025Global Convergence of Continual Learning on Non-IID Datacited by 3
- 2025Heads collapse, features stay: Why Replay needs big bufferscited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.