mlsanityData to 7 Oct 2026

Direct Multi-Turn Preference Optimization for Language Agents

arXiv 2406.14868 ↗First posted 21 Jun 2024cs.CL cs.LGcited by 63

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss. The code is available at https://github.com/swt-user/DMPO.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 51 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

24 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  9. 2023Cognitive Architectures for Language Agentscited by 422
  10. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  11. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  12. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
11 more
  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  4. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  6. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  7. 2024PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Samplingcited by 32
  8. 2023Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoningcited by 30
  9. 2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Modelscited by 26
  10. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  11. 2024MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertisecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 32; 2026: 30.

63 papers.

Sort:

By area:

  1. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  3. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  4. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  5. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  6. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  9. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agentscited by 29
  10. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  11. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvementcited by 19
  12. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
38 more
  1. 2026Benchmarking LLM Tool-Use in the Wildcited by 15
  2. 2025SAND: Boosting LLM Agents with Self-Taught Action Deliberationcited by 14
  3. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  4. 2025World Modelling Improves Language Model Agentscited by 12
  5. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  6. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 9
  7. 2025Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agentscited by 9
  8. 2025DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Modelscited by 8
  9. 2026ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialoguecited by 8
  10. 2026Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interactioncited by 7
  11. 2026LiteCoder-Terminal: Scaling Long-Horizon Terminal Environments for Learning Language Agentscited by 7
  12. 2025Learning from Failures in Multi-Attempt Reinforcement Learningcited by 6
  13. 2025Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Modelscited by 6
  14. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 6
  15. 2026TimeWarp: Evaluating Web Agents by Revisiting the Pastcited by 6
  16. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  17. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5
  18. 2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 5
  19. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  20. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  21. 2026From History to State: Constant-Context Skill Learning for LLM Agentscited by 5
  22. 2025C3-Bench: The Things Real Disturbing LLM based Agent in Multi-Taskingcited by 4
  23. 2025PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awarenesscited by 4
  24. 2025DEPO: Dual-Efficiency Preference Optimization for LLM Agentscited by 4
  25. 2026Healthcare AI GYM for Medical Agentscited by 4
  26. 2025Efficient Multi-Agent System Training with Data Influence-Oriented Tree Searchcited by 3
  27. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  28. 2025Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logscited by 3
  29. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  30. 2026Self-evolving LLM agents with in-distribution Optimizationcited by 3
  31. 2026Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectoriescited by 3
  32. 2025Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimizationcited by 2
  33. 2025Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHFcited by 2
  34. 2026DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agentscited by 2
  35. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 2
  36. 2026AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectoriescited by 2
  37. 2025Multi-Agent Synergy-Driven Iterative Visual Narrative Synthesiscited by 1
  38. 2026From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimizationcited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  2. 2024Minor DPO reject penalty to increase training robustness
  3. 2024Entropy Controllable Direct Preference Optimization
  4. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  5. 2024Direct Preference Optimization with an Offset
  6. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization
  7. 2025A Principled Loss Function for Direct Language Model Alignment
  8. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model