mlsanityData to 14 Oct 2026

ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems

arXiv 2601.11854 ↗First posted 17 Jan 2026cs.CL cs.AI cs.MAcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agentic task-oriented dialogue (TOD) requires systems to track concurrent goals, dependencies, and long-horizon state. We examine goal-lifecycle recovery from fixed dialogue trajectories. ATOD contains 1,000 synthetic dialogues annotated for six advanced-TOD properties, and ATOD-Eval defines metrics for dependency-sensitive completion, memory recall, and proactivity. We implement a symbolic-vector memory evaluator for ATOD-Eval. Among five prompt-only predictors and six matched-backbone memory baselines, our evaluator is the only configuration above 90% in both goal detection F1 and conditional status accuracy on medium dialogues. On complex dialogues, no baseline exceeds it on both metrics; it has the highest conditional status accuracy within the memory-based block and the lowest measured per-turn latency. These experiments assess lifecycle tracking rather than interactive agent task success. A cross-family judge swap and a manual audit with 94.0% agreement provide initial checks on measurement reliability. Code and data will be released at https://github.com/amazon-science/ATOD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 31 of its 57 references that are in the map. First posted on arXiv 17 Jan 2026.

Builds on

31 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents11 Nov
  2. 2024Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems15 Nov
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  4. 2024Large Language Models as Zero-shot Dialogue State Tracker through Function Calling16 Feb

Other references · 27

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  7. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  10. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  11. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  12. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
15 more
  1. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 97
  2. 2024Towards Lifelong Dialogue Agents via Timeline-based Memory Managementcited by 51
  3. 2021Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systemscited by 47
  4. 2020RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systemscited by 36
  5. 2023Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curationcited by 23
  6. 2022User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systemscited by 20
  7. 2024SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Trackingcited by 17
  8. 2022Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulatorcited by 15
  9. 2025T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planningcited by 14
  10. 2025TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisonscited by 10
  11. 2024LUCID: LLM-Generated Utterances for Complex and Interesting Dialoguescited by 7
  12. 2024HierTOD: A Task-Oriented Dialogue System Driven by Hierarchical Goalscited by 6
  13. 2025AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AIcited by 6
  14. 2019Domain-Independent turn-level Dialogue Quality Evaluation via User Satisfaction Estimationcited by 5
  15. 2023Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents
  2. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
  3. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
  4. 2025ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents
  5. 2025When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
  6. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  7. 2022Towards a Progression-Aware Autonomous Dialogue Agent
  8. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments