mlsanityData to 14 Oct 2026

Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

arXiv 2503.12854 ↗First posted 17 Mar 2025cs.CLcited by 37

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as a critical component for enhancing reasoning. However, the substantial computational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such as Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitating self-improvement for LLMs through iterative preference-based learning. We demonstrate that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly for strong base model. Furthermore, we design an iterative enhancement framework for both the generator and the reward model (RM), enabling their mutual improvement through online interaction across multiple rounds of DPO. Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower computational overhead. These findings highlight DPO as a scalable and cost-effective alternative to RL, offering a practical solution for enhancing LLM reasoning in resource-constrained situations.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 72 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

39 of 72 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Better Process Supervision with Bi-directional Rewarding Signals6 Mar
  2. 2025Process-based Self-Rewarding Language Models5 Mar
  3. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  4. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  8. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  9. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  10. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb

Other references · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2025s1: Simple test-time scalingcited by 1,341
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2024DeepSeek-V3 Technical Reportcited by 811
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2025LIMO: Less is More for Reasoningcited by 498
  12. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
17 more
  1. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  2. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  4. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  5. 2024LLM Critics Help Catch LLM Bugscited by 147
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  7. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  8. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  9. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 94
  10. 2025Self-rewarding correction for mathematical reasoningcited by 70
  11. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 63
  12. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  13. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  14. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  15. 2024LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedbackcited by 41
  16. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
  17. 2024Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 15.

37 papers.

Sort:

By area:

  1. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  2. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  3. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  4. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 34
  5. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  6. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  7. 2025Mathesis: Towards Formal Theorem Proving from Natural Languagescited by 18
  8. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  9. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  10. 2025ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learningcited by 13
  11. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 12
  12. 2025Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoningcited by 8
25 more
  1. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  2. 2025RLAE: Reinforcement Learning-Assisted Ensemble for LLMscited by 7
  3. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7
  4. 2025SparseRM: A Lightweight Preference Modeling with Sparse Autoencodercited by 7
  5. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 6
  6. 2025Enhancing Large Language Models through Structured Reasoningcited by 5
  7. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
  8. 2025Sample-efficient LLM Optimization with Reset Replaycited by 4
  9. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  10. 2025Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixingcited by 2
  11. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  12. 2025RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learningcited by 1
  13. 2026Spec-o3: A Tool-Augmented Vision-Language Agent for Rare Celestial Object Candidate Vetting via Automated Spectral Inspectioncited by 1
  14. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learningcited by 1
  15. 2026Confidence-Aware Alignment Makes Reasoning LLMs More Reliablecited by 1
  16. 2026GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zerocited by 1
  17. 2026Enhancing LLM Metacognition via Cognitive Pairwise Trainingcited by 1
  18. 2025Direct Preference Optimization for Adaptive Concept-based Explanationscited by 0
  19. 2025Bradley-Terry Policy Optimization for Generative Preference Modelingcited by 0
  20. 2026Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?cited by 0
  21. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  22. 2026LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Modelscited by 0
  23. 2026How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVRcited by 0
  24. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimizationcited by 0
  25. 2026MERGED: Multimodal Entity Resolution via Generated Expert Reasoning Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  3. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  4. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  5. 2025MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning
  6. 2025Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values
  7. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  8. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization