mlsanityData to 14 Oct 2026

Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values

arXiv 2502.13723 ↗First posted 19 Feb 2025cs.CL cs.AIcited by 6

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We introduce Direct Value Optimization (DVO), an innovative reinforcement learning framework for enhancing large language models in complex reasoning tasks. Unlike traditional methods relying on preference labels, DVO utilizes value signals at individual reasoning steps, optimizing models via a mean squared error loss. The key benefit of DVO lies in its fine-grained supervision, circumventing the need for labor-intensive human annotations. Target values within the DVO are estimated using either Monte Carlo Tree Search or an outcome value model. Our empirical analysis on both mathematical and commonsense reasoning tasks shows that DVO consistently outperforms existing offline preference optimization techniques, even with fewer training steps. These findings underscore the importance of value signals in advancing reasoning capabilities and highlight DVO as a superior methodology under scenarios lacking explicit human preference information.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 58 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

43 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs26 Jun
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold20 Jun
  3. 2024Step-level Value Preference Optimization for Mathematical Reasoning16 Jun
  4. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning1 May
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  6. 2023Llemma: An Open Language Model For Mathematics16 Oct
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug

Other references · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
24 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  3. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  4. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  8. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  9. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  10. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  11. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  12. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  13. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  14. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  15. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  16. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 159
  17. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  18. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  19. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  20. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  21. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  22. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  23. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  24. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  3. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 14
  4. 2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 7
  5. 2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Futurecited by 5
  6. 2026Segment-Aligned Policy Optimization for Multi-Modal Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
  2. 2026Continuous-Utility Direct Preference Optimization
  3. 2025A Survey of Direct Preference Optimization
  4. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  5. 2025Preference Distillation via Value based Reinforcement Learning
  6. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  7. 2025Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
  8. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey