mlsanityData to 14 Oct 2026

Reinforce LLM Reasoning through Multi-Agent Reflection

arXiv 2506.08379 ↗First posted 10 Jun 2025cs.LG cs.AI cs.CLcited by 27

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Leveraging more test-time computation has proven to be an effective way to boost the reasoning capabilities of large language models (LLMs). Among various methods, the verify-and-improve paradigm stands out for enabling dynamic solution exploration and feedback incorporation. However, existing approaches often suffer from restricted feedback spaces and lack of coordinated training of different parties, leading to suboptimal performance. To address this, we model this multi-turn refinement process as a Markov Decision Process and introduce DPSDP (Direct Policy Search by Dynamic Programming), a reinforcement learning algorithm that trains an actor-critic LLM system to iteratively refine answers via direct preference learning on self-generated data. Theoretically, DPSDP can match the performance of any policy within the training distribution. Empirically, we instantiate DPSDP with various base models and show improvements on both in- and out-of-distribution benchmarks. For example, on benchmark MATH 500, majority voting over five refinement steps increases first-turn accuracy from 58.2% to 63.2% with Ministral-based models. An ablation study further confirms the benefits of multi-agent collaboration and out-of-distribution generalization.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 62 of its 91 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

62 of 91 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MALT: Improving Reasoning with Multi-Agent LLM Training2 Dec
  2. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning18 Sep
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May
  5. 2023LEVER: Learning to Verify Language-to-Code Generation with Execution16 Feb

Other references · 57

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
38 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  4. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  6. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  8. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  9. 2023Language Models can Solve Computer Taskscited by 372
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  11. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  12. 2022Generating Sequences by Learning to Self-Correctcited by 277
  13. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  14. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  15. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  16. 2024Iterative Reasoning Preference Optimizationcited by 246
  17. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  18. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  19. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  20. 2023Nash Learning from Human Feedbackcited by 224
  21. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  22. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  23. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  24. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  25. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  26. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  27. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  28. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  29. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  30. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  31. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  32. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  33. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  34. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  35. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  36. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 67
  37. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  38. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 63

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 14.

27 papers.

Sort:

By area:

  1. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 13
  2. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  3. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  4. 2025Interactive Learning for LLM Reasoningcited by 5
  5. 2025LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMscited by 5
  6. 2025A Survey on AgentOps: Categorization, Challenges, and Future Directionscited by 4
  7. 2025Reducing Cognitive Overhead in Tool Use via Multi-Small-Agent Reinforcement Learningcited by 3
  8. 2025Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Promptingcited by 3
  9. 2026Harnessing Generalist Agents for Contextualized Time Seriescited by 3
  10. 2025Unveiling the Latent Directions of Reflection in Large Language Modelscited by 2
  11. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  12. 2026When Do Multi-Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systemscited by 2
15 more
  1. 2026PseudoAct: Leveraging Pseudocode Synthesis for Flexible Planning and Action Control in Large Language Model Agentscited by 2
  2. 2026Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Frameworkcited by 2
  3. 2026Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoningcited by 2
  4. 2025Before Humans Join the Team: Diagnosing Coordination Failures in Healthcare Robot Team Simulationcited by 1
  5. 2025AGI Requires a Coordination Layer on Top of Pattern Repositoriescited by 1
  6. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov Statescited by 1
  7. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  8. 2026CyberJurors: A Multi-Agent Simulation Task for E-Commerce Disputes Verdictcited by 1
  9. 2025TASER: Table Agents for Schema-guided Extraction and Recommendationcited by 0
  10. 2025MetaSynth: Multi-Agent Metadata Generation from Implicit Feedback in Black-Box Systemscited by 0
  11. 2025Convergence of Outputs When Two Large Language Models Interact in a Multi-Agentic Setupcited by 0
  12. 2026General learned delegation by clonescited by 0
  13. 2026Refining Over Resampling: Test-Time Self-Correction for LLM Reasoningcited by 0
  14. 2026CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agentscited by 0
  15. 2026SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
  2. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
  3. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration
  4. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  6. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
  7. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  8. 2026Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution