Data to 7 Oct 2026

Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

arXiv 2510.18874 ↗First posted 21 Oct 2025cs.LG cs.CLcited by 82

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Adapting language models (LMs) to new tasks via post-training carries the risk of degrading existing capabilities -- a phenomenon classically known as catastrophic forgetting. In this paper, toward identifying guidelines for mitigating this phenomenon, we systematically compare the forgetting patterns of two widely adopted post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a consistent trend across LM families (Llama, Qwen) and tasks (instruction following, general knowledge, and arithmetic reasoning): RL leads to less forgetting than SFT while achieving comparable or higher target task performance. To investigate the cause for this difference, we consider a simplified setting in which the LM is modeled as a mixture of two distributions, one corresponding to prior knowledge and the other to the target task. We identify that the mode-seeking nature of RL, which stems from its use of on-policy data, enables keeping prior knowledge intact when learning the target task. We then verify this insight by demonstrating that the use on-policy data underlies the robustness of RL to forgetting in practical settings, as opposed to other algorithmic choices such as the KL regularization or advantage estimation. Lastly, as a practical implication, our results highlight the potential of mitigating forgetting using approximately on-policy data, which can be substantially more efficient to obtain than fully on-policy data.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 21 Oct 2025.

Builds on

36 of 42 references.

References · 36

2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2024The Llama 3 Herd of Modelscited by 14,192
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2020Measuring Massive Multitask Language Understandingcited by 7,768
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
2016Overcoming catastrophic forgetting in neural networkscited by 4,718
2024Qwen2.5 Technical Reportcited by 4,713
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2016Learning without Forgettingcited by 2,465
2017Gradient Episodic Memory for Continual Learningcited by 1,871
2023Instruction-Following Evaluation for Large Language Modelscited by 1,143

24 more

2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
2024Continual Learning for Large Language Models: A Surveycited by 144
2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 79.

82 papers.

Sort:

By area:

2026Reinforcement Learning via Self-Distillationcited by 285
2026Self-Distillation Enables Continual Learningcited by 212
2026On-Policy Context Distillation for Language Modelscited by 161
2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
2026Online Experiential Learning for Language Modelscited by 24
2026Reinforcement World Model Learning for LLM-based Agentscited by 21
2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
2026Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillationcited by 16
2026Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoningcited by 13
2026Learning to Reason in 13 Parameterscited by 13
2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13

38 more

2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Costcited by 12
2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 11
2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capabilitycited by 11
2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 10
2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillationcited by 9
2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
2026Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Trainingcited by 7
2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Datacited by 7
2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 6
2026Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Trainingcited by 6
2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learningcited by 5
2026TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAGcited by 5
2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 5
2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 4
2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
2026Can RL Improve Generalization of LLM Agents? An Empirical Studycited by 4
2026LiteGUI: Distilling Compact GUI Agents with Reinforcement Learningcited by 4
2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weightingcited by 3
2026Self-Policy Distillation via Capability-Selective Subspace Projectioncited by 3
2026A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RLcited by 3
2026Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Treescited by 2
2026GIFT: Reconciling Post-Training Objectives via Variational Finite-Temperature Gibbs Initializationcited by 2
2026Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Modelscited by 2
2026Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilitiescited by 2
2026Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retentioncited by 2
2026Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Trainingcited by 2
2026Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategiescited by 2
2026When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizoncited by 2
2026RL Forgets! Towards Continual Policy Optimizationcited by 2

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way