mlsanityData to 14 Oct 2026

Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning

arXiv 2505.17988 ↗First posted 23 May 2025cs.LG cs.AIcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

R1-style Reinforcement Learning (RL) significantly enhances Large Language Models' reasoning capabilities, yet the mechanism behind rule-based RL remains unclear. We found that small-scale SFT has substantial influence on RL but shows poor efficiency. To explain our observations, we propose an analytical framework and compare the efficiency of SFT and RL by measuring sample effect. Our hypothetical analysis shows the potential to improve SFT efficiency. Guided by our analysis, we propose Re-distillation, a technique that aims to boost the effectiveness of small-scale distillation by sampling from the RL-trained policy. Re-distillation shows consistent surprising efficiency on three datasets and both Qwen&Llama models: Re-distilled models matched RL performance with far fewer samples and less computation. As a result, on K&K dataset, our re-distilled Qwen-2.5-1.5B model surpasses DeepSeek-V3-0324 with only 1K SFT samples. We demonstrate that re-distillation can be used to efficiently balance multiple goals in RL. Our work explains several interesting phenomena in R1-style RL, shedding light on the mechanisms behind its empirical success. Code is available at: https://github.com/on1262/deep-reasoning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 48 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

35 of 48 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  6. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  7. 2025LIMR: Less is More for RL Scaling17 Feb
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
13 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  6. 2025LIMO: Less is More for Reasoningcited by 498
  7. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  8. 2025Competitive Programming with Large Reasoning Modelscited by 119
  9. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  10. 2024Learning Dynamics of LLM Finetuningcited by 95
  11. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 93
  12. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  13. 2024The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  2. 2025Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidancecited by 3
  3. 2026CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  3. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
  4. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  5. 2025Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning
  6. 2025RL's Razor: Why Online Reinforcement Learning Forgets Less
  7. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  8. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy