mlsanityData to 14 Oct 2026

RL Fine-Tuning Heals OOD Forgetting in SFT

arXiv 2509.12235 ↗First posted 8 Sep 2025cs.LG cs.AIcited by 31

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) is a standard post-training recipe for improving Large Language Models (LLM) reasoning, but why it works remains unclear. We revisit the common claim that ``SFT memorizes, RL generalizes'' through checkpoint-wise analyses of in-distribution (ID) and out-of-distribution (OOD) reasoning. We find that OOD performance often peaks early during SFT and then declines despite continued improvement in ID reasoning. RL typically does not surpass this early SFT peak; rather, it restores OOD capability lost during later SFT, and only from a bounded range of SFT checkpoints. Further spectral analysis shows that this forgetting-and-recovery pattern correlates with rotations of singular vectors, while singular values remain largely stable. These findings suggest a more precise view of post-training dynamics: SFT can forget, RL can recover, and controlling singular-vector rotation may improve OOD robustness. Code is available at hrefhttps://github.com/jinhangzhan/RLHealsSFT.githttps://github.com/jinhangzhan/RLHealsSFT.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 76 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

52 of 76 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  2. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  3. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  4. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  5. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning20 May
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025Learning to Reason at the Frontier of Learnability17 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
28 more
  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  5. 2022Language model compression with weighted low-rank factorizationcited by 195
  6. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  7. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  8. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  9. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  10. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  11. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  13. 2023Large Language Models can Learn Rulescited by 69
  14. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  15. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
  16. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 41
  17. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 37
  18. 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
  19. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 31
  20. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
  21. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  22. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 23
  23. 2022Random matrix analysis of deep neural network weight matricescited by 21
  24. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
  25. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 10
  26. 2025Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Codingcited by 7
  27. 2015The Role of Principal Angles in Subspace Classificationcited by 5
  28. 2025FanChuan: A Multilingual and Graph-Structured Benchmark For Parody Detection and Analysiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 26.

31 papers.

Sort:

By area:

  1. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  2. 2025Debunk the Myth of SFT Generalizationcited by 8
  3. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  4. 2026Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoningcited by 7
  5. 2026Continual GUI Agentscited by 7
  6. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  7. 2026(How) Learning Rates Regulate Catastrophic Overtrainingcited by 6
  8. 2026GeoRA: Geometry-Aware Low-Rank Adaptation for RLVRcited by 4
  9. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 4
  10. 2025Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learningcited by 3
  11. 2026Probing RLVR training instability through the lens of objective-level hackingcited by 3
  12. 2026Rotation-Preserving Supervised Fine-Tuningcited by 3
19 more
  1. 2026Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuningcited by 2
  2. 2026RL Forgets! Towards Continual Policy Optimizationcited by 2
  3. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  4. 2025RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Servicescited by 1
  5. 2026TMS: Trajectory-Mixed Supervision for Reward-Free, On-Policy SFTcited by 1
  6. 2026GUI-AC: Enhancing Continual Learning in GUI Agentscited by 1
  7. 2026Rewriting or Reweighting? A Geometric Account in Language Modelscited by 1
  8. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  9. 2025Less is More: Denoising Knowledge Graphs For Retrieval Augmented Generationcited by 0
  10. 2026WinDOM: Self-Family Distillation for Small-Model GUI Groundingcited by 0
  11. 2026Where Do CoT Training Gains Land in LLM based Agents?cited by 0
  12. 2026Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignmentcited by 0
  13. 2026Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffscited by 0
  14. 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-trainingcited by 0
  15. 2026Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Featurescited by 0
  16. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  17. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectivescited by 0
  18. 2026Exploration-Preserving Policy Optimizationcited by 0
  19. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
  2. 2026Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
  3. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs
  4. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
  5. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  6. 2025RL's Razor: Why Online Reinforcement Learning Forgets Less
  7. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  8. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them