mlsanityData to 7 Oct 2026

Building Math Agents with Multi-Turn Iterative Preference Learning

arXiv 2409.02392 ↗First posted 4 Sep 2024cs.LG stat.MLcited by 66

On the map: LLM reasoning → Mathematical reasoning

also in 2 more
Abstract (arXiv)

Recent studies have shown that large language models' (LLMs) mathematical problem-solving capabilities can be enhanced by integrating external tools, such as code interpreters, and employing multi-turn Chain-of-Thought (CoT) reasoning. While current methods focus on synthetic data generation and Supervised Fine-Tuning (SFT), this paper studies the complementary direct preference learning approach to further improve model performance. However, existing direct preference learning algorithms are originally designed for the single-turn chat task, and do not fully address the complexities of multi-turn reasoning and external tool integration required for tool-integrated mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn direct preference learning framework, tailored for this context, that leverages feedback from code interpreters and optimizes trajectory-level preferences. This framework includes multi-turn DPO and multi-turn KTO as specific implementations. The effectiveness of our framework is validated through training of various language models using an augmented prompt set from the GSM8K and MATH datasets. Our results demonstrate substantial improvements: a supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5% to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 86 of its 113 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

86 of 113 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning30 Jun
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs26 Jun
  3. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving18 Jun
  4. 2024Step-level Value Preference Optimization for Mathematical Reasoning16 Jun
  5. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning1 May
  6. 2024Iterative Reasoning Preference Optimization30 Apr
  7. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Math16 Feb
  8. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset15 Feb
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  10. 2024Augmenting Math Word Problems via Iterative Question Composing17 Jan
  11. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
3 more
  1. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug
  2. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning4 Jun
  3. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Models28 Oct

Other references · 71

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Let's Verify Step by Stepcited by 3,598
38 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  13. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  14. 2022PAL: Program-aided Language Modelscited by 635
  15. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  16. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  17. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  18. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  19. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  20. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  21. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
  22. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  23. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  24. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  25. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  26. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  27. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  28. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  29. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  30. 2023Nash Learning from Human Feedbackcited by 220
  31. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  32. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  33. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  34. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  35. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  36. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  37. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  38. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 41; 2026: 9.

66 papers.

Sort:

By area:

  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  4. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  5. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  6. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  7. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  8. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  9. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  10. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  11. 2025Self-rewarding correction for mathematical reasoningcited by 67
  12. 2025LLM Agents for Education: Advances and Applicationscited by 59
38 more
  1. 2025Self-Challenging Language Model Agentscited by 56
  2. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  3. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  4. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  5. 2024Entropy-Regularized Process Reward Modelcited by 38
  6. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  9. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  10. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  11. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  12. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  13. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  14. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  15. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  16. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
  17. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  18. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  19. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  20. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 22
  21. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  22. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  23. 2025Multi-Step Visual Reasoning with Visual Tokens Scaling and Verificationcited by 19
  24. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  25. 2025GNNs as Predictors of Agentic Workflow Performancescited by 17
  26. 2024Automatic Curriculum Expert Iteration for Reliable LLM Reasoningcited by 16
  27. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  28. 2025From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learningcited by 15
  29. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  30. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  31. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoningcited by 11
  32. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancementcited by 10
  33. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
  34. 2024On Extending Direct Preference Optimization to Accommodate Tiescited by 8
  35. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  36. 2025DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Modelscited by 8
  37. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  38. 2025iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Usecited by 7

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
  2. 2024Plug-and-Play Training Framework for Preference Optimization
  3. 2024Mars-PO: Multi-Agent Reasoning System Preference Optimization
  4. 2026Agentic Multi-Turn Reasoning: A Fairness Approach
  5. 2025Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learning
  6. 2024Preference Optimization for Reasoning with Pseudo Feedback
  7. 2025Self-Evolved Preference Optimization for Enhancing Mathematical Reasoning in Small Language Models
  8. 2024Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning