mlsanityData to 14 Oct 2026

Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them

arXiv 2507.10616 ↗First posted 13 Jul 2025cs.LG cs.AI cs.CLcited by 15

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforcement learning (RL) and supervised fine-tuning (SFT), but their training dynamics are poorly understood. We present a comparative analysis of RL and SFT on the same maths problems with the same model and similar hyperparameters. We find that RL yields minor in-domain gains on maths and slight degradation on knowledge-intensive benchmarks like MMLU, while both trends are more pronounced in SFT. We also analyse model parameters across checkpoints, observing that both algorithms modify query and key weights the most. Meanwhile, SFT exhibits greater updates and also affects mid-layer MLPs more, leading us to hypothesise that this may have caused the out-of-domain degradation. We therefore investigate whether freezing parts of the model during training can mitigate the reduced performance on knowledge-intensive benchmarks. However, our results are inconclusive, with benefits on GPQA:Diamond and degradation on other benchmarks. Taken together, our observations provide a preliminary indication for why RL amplifies existing capabilities, while SFT replaces old skills with new ones.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 55 references that are in the map. First posted on arXiv 13 Jul 2025.

Builds on

39 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  2. 2025Learning to Reason under Off-Policy Guidance21 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  9. 2023Let's Verify Step by Step31 May

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
18 more
  1. 2024OpenAI o1 System Cardcited by 1,244
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  5. 20242 OLMo 2 Furiouscited by 340
  6. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  7. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  8. 2025Reasoning Models Can Be Effective Without Thinkingcited by 187
  9. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  10. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  11. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
  12. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
  13. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  14. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  15. 2025Rethinking Reflection in Pre-Trainingcited by 43
  16. 2025Learning to Reason for Long-Form Story Generationcited by 34
  17. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  18. 2024Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  2. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMscited by 20
  3. 2025Revisiting LLM Reasoning via Information Bottleneckcited by 19
  4. 2026Learning to Reason in 13 Parameterscited by 14
  5. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 10
  6. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  7. 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Modelscited by 5
  8. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoningcited by 2
  9. 2026Demystifying Reinforcement Learning Post-Training of Language Modelscited by 2
  10. 2025Dataset Construction for Training LLM to Learn Analog Circuit Knowledgecited by 1
  11. 2025Zooming into Comics: Region-Aware RL Improves Fine-Grained Comic Understanding in Vision-Language Modelscited by 1
  12. 2026Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectorscited by 1
3 more
  1. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  2. 2026Panoptic Pairwise Distortion Graphcited by 0
  3. 2026TOPCELL: Topology Optimization of Standard Cell via LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
  2. 2025A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning
  3. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  4. 2024Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models
  5. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  6. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  7. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
  8. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions