mlsanityData to 7 Oct 2026

Offline Reinforcement Learning for LLM Multi-Step Reasoning

arXiv 2412.16145 ↗First posted 20 Dec 2024cs.LG cs.AI cs.CLcited by 53

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Improving the multi-step reasoning ability of large language models (LLMs) with offline reinforcement learning (RL) is essential for quickly adapting them to complex tasks. While Direct Preference Optimization (DPO) has shown promise in aligning LLMs with human preferences, it is less suitable for multi-step reasoning tasks because (1) DPO relies on paired preference data, which is not readily available for multi-step reasoning tasks, and (2) it treats all tokens uniformly, making it ineffective for credit assignment in multi-step reasoning tasks, which often come with sparse reward. In this work, we propose OREO (Offline Reasoning Optimization), an offline RL method for enhancing LLM multi-step reasoning. Building on insights from previous works of maximum entropy reinforcement learning, it jointly learns a policy model and value function by optimizing the soft Bellman Equation. We show in principle that it reduces the need to collect pairwise data and enables better credit assignment. Empirically, OREO surpasses existing offline learning methods on multi-step reasoning benchmarks, including mathematical reasoning tasks (GSM8K, MATH) and embodied agent control (ALFWorld). The approach can be extended to a multi-iteration framework when additional resources are available. Furthermore, the learned value function can be leveraged to guide the tree search for free, which can further boost performance during test time.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 68 of its 81 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

68 of 81 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Entropy-Regularized Process Reward Model15 Dec
  2. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization11 Oct
  3. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  6. 2024Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision5 Feb
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  8. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding26 Sep
  9. 2023Let's Verify Step by Step31 May
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  12. 2022Solving math word problems with process- and outcome-based feedback25 Nov
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
38 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  12. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  13. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  14. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  15. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  16. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  17. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  18. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  19. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  20. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  21. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  22. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  23. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  24. 2024Iterative Reasoning Preference Optimizationcited by 244
  25. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  26. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  27. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  28. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  29. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  30. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  31. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  32. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  33. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  34. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  35. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  36. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  37. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  38. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 17.

53 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  5. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  6. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  7. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  8. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  9. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  10. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  11. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  12. 2025Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMAcited by 23
38 more
  1. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  2. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 19
  3. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  4. 2025Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolutioncited by 18
  5. 2025Think Clearly: Improving Reasoning via Redundant Token Pruningcited by 17
  6. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  7. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  8. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  9. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  10. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
  11. 2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 9
  12. 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learningcited by 9
  13. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  14. 2025Trust but Verify! A Survey on Verification Design for Test-time Scalingcited by 8
  15. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  16. 2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Valuescited by 6
  17. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 6
  18. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 6
  19. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
  20. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  21. 2026Can RL Improve Generalization of LLM Agents? An Empirical Studycited by 4
  22. 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifierscited by 3
  23. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  24. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 2
  25. 2025Reinforcement Learning for Long-Horizon Multi-Turn Search Agentscited by 2
  26. 2026Scaling Participation in Modular AI Systemscited by 2
  27. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  28. 2026AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectoriescited by 2
  29. 2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1
  30. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 1
  31. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  32. 2025AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Modelscited by 1
  33. 2025MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoningcited by 1
  34. 2026STRUCTUREDAGENT: Planning with AND/OR Trees for Long-Horizon Web Taskscited by 1
  35. 2025SWI: Speaking with Intent in Large Language Modelscited by 0
  36. 2026Foundation Models for Medical Imaging: Status, Challenges, and Directionscited by 0
  37. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  38. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  2. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
  3. 2025HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs
  4. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  5. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  6. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  7. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  8. 2026Hint-Guided Diversified Policy Optimization for LLM Reasoning