mlsanityData to 14 Oct 2026

RL's Razor: Why Online Reinforcement Learning Forgets Less

arXiv 2509.04259 ↗First posted 4 Sep 2025cs.LGcited by 176

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle textitRL's Razor: among all ways to solve a new task, RL prefers those closest in KL to the original model.

Key facts. Cited by 176 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 56 of its 78 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

56 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training28 Jan

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  11. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  13. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  14. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  15. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  16. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  17. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  18. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  19. 2019Large Scale Incremental Learningcited by 732
  20. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  21. 2018End-to-End Incremental Learningcited by 597
  22. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  23. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  24. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  25. 2024The Platonic Representation Hypothesiscited by 382
  26. 2018Learning without Memorizingcited by 250
  27. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  28. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  29. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  30. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  31. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  32. 2017Encoder Based Lifelong Learningcited by 102
  33. 2025Self-Adapting Language Modelscited by 84
  34. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  35. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  36. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 47
  37. 2017Less-forgetful Learning for Domain Expansion in Deep Neural Networkscited by 27
  38. 2025LADDER: Self-Improving LLMs Through Recursive Problem Decompositioncited by 25

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 140.

176 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Self-Distillation Enables Continual Learningcited by 247
  3. 2026On-Policy Context Distillation for Language Modelscited by 186
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  6. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  7. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  8. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  9. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 33
  10. 2026Online Experiential Learning for Language Modelscited by 28
  11. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  12. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 24
38 more
  1. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Modelscited by 23
  2. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMscited by 20
  3. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 19
  4. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  5. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 17
  6. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
  7. 2026Learning to Reason in 13 Parameterscited by 14
  8. 2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Costcited by 14
  9. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  10. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 13
  11. 2025Reinforcement Learning on Pre-Training Datacited by 12
  12. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12
  13. 2025How to Teach Large Multimodal Models New Skillscited by 11
  14. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  15. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Modelscited by 11
  16. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Modelscited by 10
  17. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  18. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  19. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  20. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  21. 2026RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Formatcited by 9
  22. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  23. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 9
  24. 2026Hybrid Policy Distillation for LLMscited by 9
  25. 2025Variational Reasoning for Language Modelscited by 8
  26. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
  27. 2025Debunk the Myth of SFT Generalizationcited by 8
  28. 2025Towards Understanding Self-play for LLM Reasoningcited by 8
  29. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  30. 2026Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Trainingcited by 8
  31. 2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Datacited by 8
  32. 2026Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillationcited by 8
  33. 2026On-Policy Replay for Continual Supervised Fine-Tuningcited by 8
  34. 2026Continual GUI Agentscited by 7
  35. 2026CRL-VLA: Continual Vision-Language-Action Learningcited by 7
  36. 2026Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RLcited by 7
  37. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 7
  38. 2026Why Fine-Tuning Encourages Hallucinations and How to Fix Itcited by 7

50 most cited of 176 All 176 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  2. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
  3. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  4. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
  5. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  6. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  7. 2025RL Fine-Tuning Heals OOD Forgetting in SFT
  8. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions