# Spurious Forgetting in Continual Learning of Language Models

arXiv: https://arxiv.org/abs/2501.13453 · HTML: https://mlsanity.com/paper/2501.13453/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2025.

## Built on by (most cited first, 30 of 51)

- [ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model](https://mlsanity.com/paper/2502.14420/) (2025)
- [Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning](https://mlsanity.com/paper/2507.00432/) (2025)
- [Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs](https://mlsanity.com/paper/2505.16831/) (2025)
- [SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2505.02486/) (2025)
- [Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment](https://mlsanity.com/paper/2507.18631/) (2025)
- [UItron: Foundational GUI Agent with Advanced Perception and Planning](https://mlsanity.com/paper/2508.21767/) (2025)
- [RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs](https://mlsanity.com/paper/2508.16546/) (2025)
- [Continual Learning for Generative AI: From LLMs to MLLMs and Beyond](https://mlsanity.com/paper/2506.13045/) (2025)
- [Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning](https://mlsanity.com/paper/2508.09883/) (2025)
- [Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs](https://mlsanity.com/paper/2510.02833/) (2025)
- [On-Policy Replay for Continual Supervised Fine-Tuning](https://mlsanity.com/paper/2605.29495/) (2026)
- [Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner](https://mlsanity.com/paper/2506.01301/) (2025)
- [Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model](https://mlsanity.com/paper/2606.22317/) (2026)
- [GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay](https://mlsanity.com/paper/2508.04676/) (2025)
- [Can Large Language Models Master Complex Card Games?](https://mlsanity.com/paper/2509.01328/) (2025)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [Continual Safety Alignment via Gradient-Based Sample Selection](https://mlsanity.com/paper/2604.17215/) (2026)
- [KV Cache Translation across Heterogeneous Large Language Models](https://mlsanity.com/paper/2607.28979/) (2026)
- [Efficient Knowledge Graph Unlearning with Zeroth-order Information](https://mlsanity.com/paper/2508.14013/) (2025)
- [Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models](https://mlsanity.com/paper/2602.02136/) (2026)
- [A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL](https://mlsanity.com/paper/2606.02398/) (2026)
- [MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis](https://mlsanity.com/paper/2508.08275/) (2025)
- [Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs](https://mlsanity.com/paper/2510.09885/) (2025)
- [Improving Latent Reasoning in LLMs via Soft Concept Mixing](https://mlsanity.com/paper/2511.16885/) (2025)
- [Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgetting](https://mlsanity.com/paper/2510.09152/) (2025)
- [SMI: Statistical Membership Inference for Reliable Unlearned Model Auditing](https://mlsanity.com/paper/2602.01150/) (2026)
- [Talking to Yourself: Defying Forgetting in Large Language Models](https://mlsanity.com/paper/2602.20162/) (2026)
- [Attribution-Guided Continual Learning for Large Language Models](https://mlsanity.com/paper/2605.05285/) (2026)
- [Rethinking Transfer in Continual Learning: A Replay-Based Realisation](https://mlsanity.com/paper/2607.15587/) (2026)
- [Contamination Means Overestimation? A Fine-Grained Empirical Study in Code Intelligence](https://mlsanity.com/paper/2506.02791/) (2025)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.