mlsanityData to 7 Oct 2026

Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

arXiv 2406.18629 ↗First posted 26 Jun 2024cs.LG cs.AI cs.CLcited by 265

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) due to the extensive and precise chain of reasoning required for accuracy. Ensuring the correctness of each reasoning step is critical. To address this, we aim to enhance the robustness and factuality of LLMs by learning from human feedback. However, Direct Preference Optimization (DPO) has shown limited benefits for long-chain mathematical reasoning, as models employing DPO struggle to identify detailed errors in incorrect answers. This limitation stems from a lack of fine-grained process supervision. We propose a simple, effective, and data-efficient method called Step-DPO, which treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Additionally, we have developed a data construction pipeline for Step-DPO, enabling the creation of a high-quality dataset containing 10K step-wise preference pairs. We also observe that in DPO, self-generated data is more effective than data generated by humans or GPT-4, due to the latter's out-of-distribution nature. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at https://github.com/dvlab-research/Step-DPO.

Key facts. Cited by 265 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 48 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

46 of 48 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models23 May
  2. 2024Common 7B Language Models Already Possess Strong Math Capabilities7 Mar
  3. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  4. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs26 Feb
  5. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Math16 Feb
  6. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset15 Feb
  7. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning9 Feb
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  9. 2024Augmenting Math Word Problems via Iterative Question Composing17 Jan
  10. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline16 Jan
  11. 2023Llemma: An Open Language Model For Mathematics16 Oct
  12. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning5 Oct
3 more
  1. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  3. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
19 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  5. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  7. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  8. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  9. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  10. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  11. 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 225
  12. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  13. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  14. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  15. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  16. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoningcited by 69
  17. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  18. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
  19. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 170; 2026: 65.

265 papers.

Sort:

By area:

  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  6. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  7. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  8. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  9. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  10. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  11. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  12. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
38 more
  1. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  2. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  3. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  4. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  6. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 97
  7. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  8. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  9. 2024Process Reward Model with Q-Value Rankingscited by 78
  10. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  11. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  12. 2025Self-rewarding correction for mathematical reasoningcited by 67
  13. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  14. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  15. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  16. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  17. 2025FineVision: Open Data Is All You Needcited by 51
  18. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  19. 2025TreeRPO: Tree Relative Policy Optimizationcited by 45
  20. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  21. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  22. 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
  23. 2024Entropy-Regularized Process Reward Modelcited by 38
  24. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  25. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  26. 2025A Survey of Direct Preference Optimizationcited by 35
  27. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  28. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 32
  29. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  30. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  31. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agentscited by 29
  32. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  33. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 28
  34. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  35. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  36. 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agentscited by 27
  37. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  38. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25

50 most cited of 265 All 265 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
  2. 2024Step-level Value Preference Optimization for Mathematical Reasoning
  3. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning
  4. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
  5. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  6. 2026HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
  7. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  8. 2024Preference Optimization for Reasoning with Pseudo Feedback