mlsanityData to 7 Oct 2026

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

arXiv 2501.17161 ↗First posted 28 Jan 2025cs.AI cs.CV cs.LGcited by 655

Not on the map. Why?

Cited by 328 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

Key facts. Cited by 655 papers in the map (data through 7 Oct 2026). Builds on 56 of its 68 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

56 of 68 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  13. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  15. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  16. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  17. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  18. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  19. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  20. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  21. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  22. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  23. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  24. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  25. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  26. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  27. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  28. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  29. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  30. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  31. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  32. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  33. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  34. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  35. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  36. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  37. 2023Large Language Models can Learn Rulescited by 66
  38. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 373; 2026: 282.

655 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  3. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  4. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  5. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  6. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  7. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  8. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  9. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  10. 2026Self-Distillation Enables Continual Learningcited by 212
  11. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  12. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
38 more
  1. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  2. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  3. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
  6. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  7. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  8. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  9. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  10. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  11. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  12. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  13. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 116
  14. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  15. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  16. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  17. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 92
  18. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  19. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 89
  20. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  21. 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 86
  22. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 86
  23. 2025Flow Matching Policy Gradientscited by 86
  24. 2025Visual Planning: Let's Think Only with Imagescited by 85
  25. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  26. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  27. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  28. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 80
  29. 2025Agent Learning via Early Experiencecited by 79
  30. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  31. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  32. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  33. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  34. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  35. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  36. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  37. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  38. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60

50 most cited of 655 All 655 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability
  2. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
  3. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  4. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  5. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
  6. 2026Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
  7. 2025RL Fine-Tuning Heals OOD Forgetting in SFT
  8. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning