mlsanityData to 7 Oct 2026

Iterative Reasoning Preference Optimization

arXiv 2404.19733 ↗First posted 30 Apr 2024cs.CL cs.AIcited by 244

On the map: LLM reasoning → Mathematical reasoning

also in 1 more
Abstract (arXiv)

Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy on GSM8K, MATH, and ARC-Challenge for Llama-2-70B-Chat, outperforming other Llama-2-based models not relying on additionally sourced datasets. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.

Key facts. Cited by 244 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 35 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

31 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset15 Feb
  2. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023PaLM 2 Technical Reportcited by 1,154
17 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  3. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  4. 2024Self-Rewarding Language Modelscited by 654
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  11. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  12. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  13. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  14. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  15. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 68
  16. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  17. 2023Chain of Thought Prompting Elicits Knowledge Augmentationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 74; 2025: 118; 2026: 52.

244 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  2. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  4. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  7. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  9. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  10. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  11. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  12. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
38 more
  1. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  3. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  4. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  5. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  6. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  7. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  8. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  9. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  10. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  11. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  12. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  13. 2025Self-rewarding correction for mathematical reasoningcited by 67
  14. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  15. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  16. 2025Can Large Reasoning Models Self-Train?cited by 66
  17. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  18. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  19. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  20. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  21. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  22. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  23. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  24. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  25. 2024Self-Consistency Preference Optimizationcited by 50
  26. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  27. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  28. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  29. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  30. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  31. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 41
  32. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  33. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  34. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 39
  35. 2024Entropy-Regularized Process Reward Modelcited by 38
  36. 2025A Survey of Direct Preference Optimizationcited by 35
  37. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  38. 2024Direct Judgement Preference Optimizationcited by 33

50 most cited of 244 All 244 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Thinking Preference Optimization
  2. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
  3. 2025Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
  4. 2025Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
  5. 2025Evaluating GRPO and DPO for Faithful Chain-of-Thought Reasoning in LLMs
  6. 2024Preference Optimization for Reasoning with Pseudo Feedback
  7. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  8. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation