mlsanityData to 7 Oct 2026

Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization

arXiv 2410.09302 ↗First posted 11 Oct 2024cs.LG cs.AI cs.CLcited by 22

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) plays a crucial role in aligning large language models (LLMs) with human preferences and improving their ability to perform complex tasks. However, current approaches either require significant computational resources due to the use of multiple models and extensive online sampling for training (e.g., PPO) or are framed as bandit problems (e.g., DPO, DRO), which often struggle with multi-step reasoning tasks, such as math problem solving and complex reasoning that involve long chains of thought. To overcome these limitations, we introduce Direct Q-function Optimization (DQO), which formulates the response generation process as a Markov Decision Process (MDP) and utilizes the soft actor-critic (SAC) framework to optimize a Q-function directly parameterized by the language model. The MDP formulation of DQO offers structural advantages over bandit-based methods, enabling more effective process supervision. Experimental results on two math problem-solving datasets, GSM8K and MATH, demonstrate that DQO outperforms previous methods, establishing it as a promising offline reinforcement learning approach for aligning language models.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 50 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

44 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  2. 2023Let's Verify Step by Step31 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
27 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  5. 2022Self-critiquing models for assisting human evaluatorscited by 372
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  7. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  8. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  9. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  10. 2023Nash Learning from Human Feedbackcited by 220
  11. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  12. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  13. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  14. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  15. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  16. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  17. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  18. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  19. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  20. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  21. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  22. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Timecited by 40
  23. 2024Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutionscited by 15
  24. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  25. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  26. 2024Enhancing textual textbook question answering with large language models and retrieval augmented generationcited by 13
  27. 2024PSYDIAL: Personality-based Synthetic Dialogue Generation using Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 3.

22 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  4. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  6. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  7. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  8. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  9. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  10. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  11. 2025Synthetic Data RL: Task Definition Is All You Needcited by 15
  12. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
10 more
  1. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  2. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  3. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 9
  4. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  5. 2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Valuescited by 6
  6. 2025ShiQ: Bringing back Bellman to LLMscited by 3
  7. 2026Near-Optimal Regret for KL-Regularized Multi-Armed Banditscited by 3
  8. 2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1
  9. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  10. 2026rePIRL: Learn PRM with Inverse RL for LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  2. 2025Reinforcement learning fine-tuning of language model for instruction following and math reasoning
  3. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  4. 2026DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning
  5. 2026LamPO: A Lambda Style Policy Optimization for Reasoning Language Models
  6. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
  7. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  8. 2026Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning