mlsanityData to 7 Oct 2026

DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models

arXiv 2504.02882 ↗First posted 2 Apr 2025cs.CL cs.LGcited by 8

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM's dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in information gathering, 91% in tool call rejection) with substantial improvements over baseline (44% and 9.6% respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 41 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

23 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Direct Multi-Turn Preference Optimization for Language Agents21 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  9. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  10. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  11. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  12. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
7 more
  1. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  2. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  3. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 69
  4. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  5. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  6. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
  7. 2024FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
  2. 2025PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awarenesscited by 4
  3. 2026Healthcare AI GYM for Medical Agentscited by 4
  4. 2026Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectoriescited by 3
  5. 2025Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Modelscited by 2
  6. 2026FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Taskscited by 2
  7. 2026Emerging from Ground: Addressing Intent Deviation in Tool-Using Agents via Deriving Real Calls into Virtual Trajectoriescited by 0
  8. 2026Clueing up LLMs with Tool-Augmented Deductive Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
  2. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings
  3. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
  4. 2024Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment
  5. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  6. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  7. 2025Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky
  8. 2025Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction