Data to 7 Oct 2026

Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

arXiv 2403.01244 ↗First posted 2 Mar 2024cs.CL cs.AIcited by 114

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

21 of 31 references.

References · 21

2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2020Measuring Massive Multitask Language Understandingcited by 7,768
2016Overcoming catastrophic forgetting in neural networkscited by 4,718
2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
2017Gradient Episodic Memory for Continual Learningcited by 1,871
2018Experience Replay for Continual Learningcited by 1,074
2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
2023Progressive Prompts: Continual Learning for Language Modelscited by 195
2019Episodic Memory in Lifelong Language Learningcited by 188
2018Reinforced Continual Learningcited by 186

9 more

2022Fine-tuned Language Models are Continual Learnerscited by 152
2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 130
2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
2020CPR: Classifier-Projection Regularization for Continual Learningcited by 62
2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 39
2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
2022Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approachcited by 13
2019Neural Architecture Search for Class-incremental Learningcited by 10
2023Response Enhanced Semi-supervised Dialogue Query Generationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 50; 2026: 46.

114 papers.

Sort:

By area:

2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
2024A Survey on Self-Evolution of Large Language Modelscited by 86
2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
2025R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learningcited by 21
2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
2024How Much Can We Forget about Data Contamination?cited by 14
2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14

38 more

2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learningcited by 12
2025Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruningcited by 12
2025Scalable Fingerprinting of Large Language Modelscited by 11
2025Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?cited by 11
2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 10
2025How to Teach Large Multimodal Models New Skillscited by 10
2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
2025Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAGcited by 9
2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
2025Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgettingcited by 8
2025Mapping Post-Training Forgetting in Language Models at Scalecited by 8
2026A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMscited by 8
2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigationcited by 7
2025Mitigating Memorization in LLMs using Activation Steeringcited by 7
2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
2026Towards Policy-Adaptive Image Guardrail: Benchmark and Methodcited by 6
2026Can Muon Fine-tune Adam-Pretrained Models?cited by 6
2025Context-Free Synthetic Data Mitigates Forgettingcited by 5
2025ConsistentChat: Building Skeleton-Guided Consistent Multi-Turn Dialogues for Large Language Models from Scratchcited by 5
2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
2026Contextual Agentic Memory is a Memo, Not True Memorycited by 5
2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 5
2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
2025AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learningcited by 4
2025ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuningcited by 4
2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
2026MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuningcited by 4
2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Modelcited by 4
2024Unveiling and Addressing Pseudo Forgetting in Large Language Modelscited by 3
2025Reinforced Interactive Continual Learning via Real-time Noisy Human Feedbackcited by 3

50 most cited of 114 All 114 (JSON)