Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification
Not on the map. Why?
Abstract (arXiv)
Fine-tuning foundation models on new tasks inevitably suffer from catastrophic forgetting. While existing works attempt to mitigate this on the basis of parameter-efficient fine-tuning methods, they adopted an overly restrictive Subspace Orthogonality condition. In this paper, we introduce a purely post-hoc and tuning-agnostic weight rectification framework that achieves Parameter Space Orthogonality, which is the necessary and sufficient condition for preserving historical performance to the first order. By projecting parameter updates into the JAcobian NUll Space (JANUS), our method significantly recovers compromised historical knowledge without interfering with the underlying fine-tuning process. To overcome the local validity of the Jacobian approximation, we further propose a Multi-step Adaptive Rectification mechanism that utilizes the JANUS shift to dynamically verify the valid trust region and adjust step sizes. Coupled with our proposed ghost projection, ghost orientation comparison, and sequence-level singular value decomposition compression techniques, JANUS also achieves great temporal and spatial efficiency. Experiments demonstrate that JANUS seamlessly integrates with various fine-tuning methods, significantly mitigating the stability-plasticity dilemma by recovering historical knowledge while preserving downstream task adaptation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 57 references that are in the map. First posted on arXiv 17 Sep 2026.
Builds on
49 of 57 references.
References · 49
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
37 more
- 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
- 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
- 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2017Learning multiple visual domains with residual adapterscited by 557
- 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
- 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
- 2019Orthogonal Gradient Descent for Continual Learningcited by 369
- 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2021Gradient Projection Memory for Continual Learningcited by 348
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
- 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
- 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
- 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
- 2021Training Neural Networks with Fixed Sparse Maskscited by 214
- 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 201
- 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
- 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
- 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
- 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
- 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
- 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
- 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 94
- 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 93
- 2024Data Shapley in One Training Runcited by 81
- 2021AFEC: Active Forgetting of Negative Transfer in Continual Learningcited by 73
- 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51
- 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 47
- 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
- 2026KeepLoRA: Continual Learning with Residual Gradient Adaptationcited by 21
- 2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 16
- 2025Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Modelscited by 7
- 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 5
Built on by
Not cited yet.