mlsanityData to 7 Oct 2026

Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems

arXiv 2402.13374 ↗First posted 20 Feb 2024cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

32 of 50 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  12. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
20 more
  1. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  2. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  3. 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
  4. 2016A User Simulator for Task-Completion Dialoguescited by 56
  5. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
  6. 2022Description-Driven Task-Oriented Dialog Modelingcited by 50
  7. 2019How to Build User Simulators to Train RL-based Dialog Systemscited by 44
  8. 2022Controllable Dialogue Simulation with In-Context Learningcited by 42
  9. 2020Language Models as Few-Shot Learner for Task-Oriented Dialogue Systemscited by 41
  10. 2016A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systemscited by 39
  11. 2018User Modeling for Task Oriented Dialoguescited by 36
  12. 2022Achieving Reliable Human Assessment of Open-Domain Dialogue Systemscited by 31
  13. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  14. 2023User Simulation for Evaluating Information Access Systemscited by 30
  15. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 24
  16. 2021Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systemscited by 23
  17. 2023Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyondcited by 23
  18. 2022GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformerscited by 22
  19. 2023In-Context Learning User Simulators for Task-Oriented Dialog Systemscited by 19
  20. 2023EmoUS: Simulating User Emotions in Task-Oriented Dialoguescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 14.

24 papers.

Sort:

By area:

  1. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  2. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  3. 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 19
  4. 2025Non-Collaborative User Simulators for Tool Agentscited by 17
  5. 2026HorizonBench: Long-Horizon Personalization with Evolving Preferencescited by 14
  6. 2025ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Modelscited by 12
  7. 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learningcited by 8
  8. 2025Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planningcited by 7
  9. 2026Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generationcited by 7
  10. 2026Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agentscited by 6
  11. 2025DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulationcited by 5
  12. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
12 more
  1. 2026Learning User Simulators with Turing Rewardscited by 3
  2. 2025TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialoguescited by 2
  3. 2026Multi-Agent Home Energy Management Assistantcited by 2
  4. 2025LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluationcited by 1
  5. 2025From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agentscited by 1
  6. 2026SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?cited by 1
  7. 2026SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Sciencecited by 1
  8. 2025Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifestocited by 0
  9. 2026Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversationcited by 0
  10. 2026Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commercecited by 0
  11. 2026Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental Statescited by 0
  12. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0