mlsanityData to 14 Oct 2026

FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning

arXiv 2601.03938 ↗First posted 7 Jan 2026cs.LG cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) for large language models (LLMs) aims to enable sequential knowledge acquisition without catastrophic forgetting. Memory replay methods are widely used for their practicality and effectiveness, but most rely on fixed, step-based heuristics that often misalign with the model's actual learning progress, since identical training steps can result in varying degrees of parameter change. Motivated by recent findings that LLM forgetting mirrors the Ebbinghaus human forgetting curve, we propose FOREVER (FORgEtting curVe-inspired mEmory Replay), a novel CL framework that aligns replay schedules with a model-centric notion of time. FOREVER defines model time using the magnitude of optimizer updates, allowing forgetting curve-inspired replay intervals to align with the model's internal evolution rather than raw training steps. Building on this approach, FOREVER incorporates a forgetting curve-based replay scheduler to determine when to replay and an intensity-aware regularization mechanism to adaptively control how to replay. Extensive experiments on three CL benchmarks and models ranging from 0.6B to 13B parameters demonstrate that FOREVER consistently mitigates catastrophic forgetting.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

46 of 71 references.

References · 46

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  6. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  7. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  9. 2024Continual Learning with Pre-Trained Models: A Surveycited by 154
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  11. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 124
  12. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 118
34 more
  1. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
  2. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  3. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  4. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 60
  5. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  6. 2024Towards Lifelong Dialogue Agents via Timeline-based Memory Managementcited by 51
  7. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  8. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  9. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 30
  10. 2025Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Mergingcited by 27
  11. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  12. 2024Unlocking Continual Learning Abilities in Language Modelscited by 23
  13. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  14. 2025You Don't Need Pre-built Graphs for RAG: Retrieval Augmented Generation with Adaptive Reasoning Structurescited by 23
  15. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  16. 2024TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidationcited by 18
  17. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  18. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 15
  19. 2024Parenting: Optimizing Knowledge Selection of Retrieval-Augmented Language Models with Parameter Decoupling and Tailored Tuningcited by 14
  20. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  21. 2025Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruningcited by 13
  22. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 10
  23. 2025Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspectivecited by 10
  24. 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptationcited by 10
  25. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  26. 2025Continuous Subspace Optimization for Continual Learningcited by 9
  27. 2024Continual Dialogue State Tracking via Reason-of-Select Distillationcited by 8
  28. 2023Sub-network Discovery and Soft-masking for Continual Learning of Mixed Taskscited by 7
  29. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  30. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  31. 2025Do Your Best and Get Enough Rest for Continual Learningcited by 5
  32. 2025RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Mergingcited by 5
  33. 2024An Efficient Rehearsal Scheme for Catastrophic Forgetting Mitigation during Multi-stage Fine-tuningcited by 4
  34. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Trainingcited by 5
  2. 2026FSFM: A Biologically-Inspired Framework for Selective Forgetting of Agent Memorycited by 3
  3. 2026Not All Memories Age the Same: Autodiscovery of Adaptive Decay in Knowledge Graphscited by 0
  4. 2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Modelscited by 0
  5. 2026MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Modelscited by 0
  6. 2026Continual Learning in Transitioncited by 0
  7. 2026SAVOR: Self-Aware Visual Grounding via Confidence-Calibrated Reinforcement Learning for Multimodal Hallucination Mitigationcited by 0
  8. 2026Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretrainingcited by 0