mlsanityData to 7 Oct 2026

Spurious Forgetting in Continual Learning of Language Models

arXiv 2501.13453 ↗First posted 23 Jan 2025cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jan 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 24.

47 papers.

Sort:

By area:

  1. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  2. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  3. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  4. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  5. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  6. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  7. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  8. 2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoningcited by 11
  9. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  10. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  11. 2025Can Large Language Models Master Complex Card Games?cited by 4
  12. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
35 more
  1. 2026Continual Safety Alignment via Gradient-Based Sample Selectioncited by 4
  2. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Modelcited by 4
  3. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  4. 2026Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Modelscited by 3
  5. 2026A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RLcited by 3
  6. 2025GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replaycited by 2
  7. 2025MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosiscited by 2
  8. 2025Efficient Knowledge Graph Unlearning with Zeroth-order Informationcited by 2
  9. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2
  10. 2025Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgettingcited by 1
  11. 2025Improving Latent Reasoning in LLMs via Soft Concept Mixingcited by 1
  12. 2026SMI: Statistical Membership Inference for Reliable Unlearned Model Auditingcited by 1
  13. 2026Talking to Yourself: Defying Forgetting in Large Language Modelscited by 1
  14. 2025Contamination Means Overestimation? A Fine-Grained Empirical Study in Code Intelligencecited by 0
  15. 2025BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understandingcited by 0
  16. 2025Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis and Interpretationcited by 0
  17. 2025SCALE: Upscaled Continual Learning of Large Language Modelscited by 0
  18. 2025Real Time Detection and Quantitative Analysis of Spurious Forgetting in Continual Learningcited by 0
  19. 2026FIT to Forget: Robust Continual Unlearning for Large Language Modelscited by 0
  20. 2026Awakening the Sleeping Agent: Lean-Specific Agentic Data Reactivates General Tool Use in Goedel Provercited by 0
  21. 2026Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuningcited by 0
  22. 2026Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelinescited by 0
  23. 2026Attribution-Guided Continual Learning for Large Language Modelscited by 0
  24. 2026Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learningcited by 0
  25. 2026Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timecited by 0
  26. 2026When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shiftcited by 0
  27. 2026CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflictscited by 0
  28. 2026Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuningcited by 0
  29. 2026Rethinking Transfer in Continual Learning: A Replay-Based Realisationcited by 0
  30. 2026TopoTuner: Topological Finetuning of Large Language Modelscited by 0
  31. 2026Continual Learning in Transitioncited by 0
  32. 2026Calibration-Preserving Pruning: Compression as a Reliability Contractcited by 0
  33. 2026Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspectivecited by 0
  34. 2026Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Followingcited by 0
  35. 2026A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speakscited by 0