# SR2-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning

arXiv: https://arxiv.org/abs/2605.07420 · HTML: https://mlsanity.com/paper/2605.07420/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 48 of its 74 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning](https://mlsanity.com/paper/2205.13218/) (2022)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [Snapshot Distillation: Teacher-Student Optimization in One Generation](https://mlsanity.com/paper/1812.00123/) (2018)
- [PyCIL: A Python Toolbox for Class-Incremental Learning](https://mlsanity.com/paper/2112.12533/) (2021)
- [Theory on Forgetting and Generalization of Continual Learning](https://mlsanity.com/paper/2302.05836/) (2023)
- [R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning](https://mlsanity.com/paper/2203.13104/) (2022)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Theory on Mixture-of-Experts in Continual Learning](https://mlsanity.com/paper/2406.16437/) (2024)

## Abstract (arXiv)

Pre-trained models with parameter-efficient fine-tuning (PEFT) have demonstrated promising potential for class-incremental learning (CIL), yet catastrophic forgetting still persists when adapting models to new tasks. In this paper, we present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift, i.e., the progressive disruption of relationships among layer-wise representations during the learning of new tasks. We theoretically show that the increase of such drift reduces the classification margins of previously learned tasks, thereby degrading overall model performance. To address this issue, we propose underlineSelf-underlineRectifying inter-layer underlineRelation Low-Rank Adaptation~(SR2-LoRA), a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift. Specifically, SR2-LoRA constructs the relation matrices induced by the previous and current models on current-task samples, and aligns the corresponding singular values. We further theoretically show that this alignment exhibits greater robustness to estimation perturbations than direct entry-wise alignment. Extensive experiments on standard CIL benchmarks demonstrate that SR2-LoRA effectively mitigates catastrophic forgetting, with its advantages becoming more pronounced as the number of tasks increases. Code is available in the hrefhttps://github.com/FqWan24/SR-2-LoRArepository.