mlsanityData to 7 Oct 2026

Agentic Multi-Turn Reasoning: A Fairness Approach

arXiv 2609.33323 ↗First posted 27 Sep 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code also fits: Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have enabled agentic systems capable of solving complex tasks through multi-turn planning, tool use, verification, and memory updates. However, learning agentic systems remains difficult due to two fundamental challenges, i.e., (1) long-horizon credit assignment, where supervision is available only at the final outcome, and (2) imbalanced data distributions, where dominant data patterns bias optimization and weaken adaptation to rare but informative reasoning behaviors. In this paper, we propose Fair Multi-Level Preference Optimization (Fair-MPO or Phi-MPO), a new preference optimization framework for agentic learning. We first show that Multi-Level Preference Optimization provides a principled and more computationally efficient framework for long-horizon reasoning. Then, we introduce a Fair Multi-Level Objective that addresses imbalance in agentic learning. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both long-horizon reasoning and data imbalance. Our experiments on agentic reasoning benchmarks demonstrate that our approach achieves State-of-the-Art (SOTA) performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 37 of its 65 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

37 of 65 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use1 Sep
  2. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning22 May
  3. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving12 May
  4. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning25 Apr
  5. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently21 Apr
  6. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  7. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning16 Feb
  8. 2024MALT: Improving Reasoning with Multi-Agent LLM Training2 Dec

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  12. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
17 more
  1. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  2. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  3. 2025Learning to Reason under Off-Policy Guidancecited by 266
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  5. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  6. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  7. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  8. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  9. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 132
  10. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  11. 2026XSkill: Continual Learning from Experience and Skills in Multimodal Agentscited by 71
  12. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  13. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  14. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 25
  15. 2025TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluationcited by 9
  16. 2025PE-MA: Parameter-Efficient Co-Evolution of Multi-Agent Systemscited by 2
  17. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mars-PO: Multi-Agent Reasoning System Preference Optimization
  2. 2024Building Math Agents with Multi-Turn Iterative Preference Learning
  3. 2025AlphaApollo: A System for Deep Agentic Reasoning
  4. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
  5. 2026Unifying Temporal and Structural Credit Assignment in LLM-Based Multi-Agent Prompt Optimization
  6. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debate
  7. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
  8. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment