mlsanityData to 14 Oct 2026

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

arXiv 2411.09972 ↗First posted 15 Nov 2024cs.CL cs.AIcited by 11

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Traditionally, offline datasets have been used to evaluate task-oriented dialogue (TOD) models. These datasets lack context awareness, making them suboptimal benchmarks for conversational systems. In contrast, user-agents, which are context-aware, can simulate the variability and unpredictability of human conversations, making them better alternatives as evaluators. Prior research has utilized large language models (LLMs) to develop user-agents. Our work builds upon this by using LLMs to create user-agents for the evaluation of TOD systems. This involves prompting an LLM, using in-context examples as guidance, and tracking the user-goal state. Our evaluation of diversity and task completion metrics for the user-agents shows improved performance with the use of better prompts. Additionally, we propose methodologies for the automatic evaluation of TOD models within this dynamic framework.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 36 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

26 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agents2 Feb
  2. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  12. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
12 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  2. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  3. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  4. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  5. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  6. 2023PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completioncited by 30
  7. 2022GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformerscited by 23
  8. 2023User Simulation with Large Language Models for Evaluating Task-Oriented Dialoguecited by 19
  9. 2022Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulatorcited by 15
  10. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  11. 2022Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systemscited by 9
  12. 2021Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialoguecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 48
  2. 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 21
  3. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 19
  4. 2026Agent-as-a-Judgecited by 17
  5. 2025clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisationscited by 4
  6. 2025Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversationscited by 2
  7. 2025Deflanderization for Game Dialogue: Balancing Character Authenticity with Task Execution in LLM-based NPCscited by 0
  8. 2026ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systemscited by 0
  9. 2026CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulatorscited by 0
  10. 2026DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactionscited by 0
  11. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Human vs. Agent in Task-Oriented Conversations
  2. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration
  3. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
  4. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  5. 2026Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents
  7. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  8. 2024Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models