mlsanityData to 14 Oct 2026

ReFT: Reasoning with Reinforced Fine-Tuning

arXiv 2401.08967 ↗First posted 17 Jan 2024cs.CLcited by 239

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

One way to enhance the reasoning capability of Large Language Models (LLMs) is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT) annotations. This approach does not show sufficiently strong generalization ability, however, because the training only relies on the given CoT data. In math problem-solving, for example, there is usually only one annotated reasoning path for each question in the training data. Intuitively, it would be better for the algorithm to learn from multiple annotated reasoning paths given a question. To address this issue, we propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of learning LLMs for reasoning, with math problem-solving as an example. ReFT first warmups the model with SFT, and then employs on-line reinforcement learning, specifically the PPO algorithm in this paper, to further fine-tune the model, where an abundance of reasoning paths are automatically sampled given the question and the rewards are naturally derived from the ground-truth answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that ReFT significantly outperforms SFT, and the performance can be potentially further boosted by combining inference-time strategies such as majority voting and re-ranking. Note that ReFT obtains the improvement by learning from the same training questions as SFT, without relying on extra or augmented training questions. This indicates a superior generalization ability for ReFT.

Key facts. Cited by 239 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 41 of its 64 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

41 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning1 Aug

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
28 more
  1. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  7. 2022Galactica: A Large Language Model for Sciencecited by 744
  8. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  9. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  13. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  14. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  15. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  16. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  17. 2018Online Learning: A Comprehensive Surveycited by 205
  18. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  19. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  20. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  21. 2019Revisiting Self-Training for Neural Sequence Generationcited by 155
  22. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  23. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 78
  24. 2018Reinforcement Learning from Imperfect Demonstrationscited by 75
  25. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  26. 2023Interpretable Math Word Problem Solution Generation Via Step-by-step Planningcited by 17
  27. 2023Design of Chain-of-Thought in Math Problem Solvingcited by 12
  28. 2023Leveraging Training Data in Few-Shot Prompting for Numerical Reasoningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 150; 2026: 60.

239 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  2. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  3. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  5. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  6. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  7. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  8. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 154
  9. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  10. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  12. 2024A Survey on Large Language Model-Based Game Agentscited by 112
38 more
  1. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  2. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  3. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 93
  4. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 90
  5. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  6. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  7. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 83
  8. 2025Concise Reasoning via Reinforcement Learningcited by 79
  9. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  10. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  11. 2025Safety in Large Reasoning Models: A Surveycited by 64
  12. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  13. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  14. 2025Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Modelscited by 62
  15. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  16. 2025One Token to Fool LLM-as-a-Judgecited by 60
  17. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  18. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  19. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  20. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 49
  21. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 49
  22. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  23. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 47
  24. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  25. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  26. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  27. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  28. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  29. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  30. 2025ARM: Adaptive Reasoning Modelcited by 39
  31. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  32. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  33. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
  34. 2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Financecited by 35
  35. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  36. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  37. 2025Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lenscited by 34
  38. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewardingcited by 32

50 most cited of 239 All 239 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning
  2. 2025Thinking Preference Optimization
  3. 2026Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworks
  4. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?
  5. 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models
  6. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning
  7. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
  8. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions