# DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues

arXiv: https://arxiv.org/abs/2405.13028 · HTML: https://mlsanity.com/paper/2405.13028/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems](https://mlsanity.com/paper/1805.06966/) (2018)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [User Modeling for Task Oriented Dialogues](https://mlsanity.com/paper/1811.04369/) (2018)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022)
- [In-Context Learning User Simulators for Task-Oriented Dialog Systems](https://mlsanity.com/paper/2306.00774/) (2023)

## Built on by (most cited first, 24 of 24)

- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants](https://mlsanity.com/paper/2506.09902/) (2025)
- [HumanLM: Simulating Users with State Alignment Beats Response Imitation](https://mlsanity.com/paper/2603.03303/) (2026)
- [Goal Alignment in LLM-Based User Simulators for Conversational AI](https://mlsanity.com/paper/2507.20152/) (2025)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends](https://mlsanity.com/paper/2409.14195/) (2024)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents](https://mlsanity.com/paper/2605.12894/) (2026)
- [SPHERE: An Evaluation Card for Human-AI Systems](https://mlsanity.com/paper/2504.07971/) (2025)
- [Human vs. Agent in Task-Oriented Conversations](https://mlsanity.com/paper/2509.17619/) (2025)
- [SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation](https://mlsanity.com/paper/2510.11997/) (2025)
- [DiscoverLLM: From Executing Intents to Discovering Them](https://mlsanity.com/paper/2602.03429/) (2026)
- [Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors](https://mlsanity.com/paper/2605.07847/) (2026)
- [DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation](https://mlsanity.com/paper/2512.20773/) (2025)
- [Game Development as Human-LLM Interaction](https://mlsanity.com/paper/2408.09386/) (2024)
- [Sell More, Play Less: Benchmarking LLM Realistic Selling Skill](https://mlsanity.com/paper/2604.07054/) (2026)
- [Instruction Tuning with Retrieval-based Examples Ranking for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2405.18035/) (2024)
- [How to Train a Real-World Silicon Concierge? Internalizing Complex Business Workflow to Only OneModel](https://mlsanity.com/paper/2608.20350/) (2026)
- [From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs](https://mlsanity.com/paper/2604.14137/) (2026)
- [CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators](https://mlsanity.com/paper/2605.08334/) (2026)
- [Prompt Optimization for User Simulation in Conversational Recommender Systems: A Multi-Objective Framework](https://mlsanity.com/paper/2607.00010/) (2026)
- [Large-Scale ChatBot Validation Through Customer Digital Twin Simulations](https://mlsanity.com/paper/2607.26060/) (2026)
- [Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems](https://mlsanity.com/paper/2608.19549/) (2026)
- [CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking](https://mlsanity.com/paper/2610.02460/) (2026)

## Abstract (arXiv)

User Simulators play a pivotal role in training and evaluating task-oriented dialogue systems. Traditional user simulators typically rely on human-engineered agendas, resulting in generated responses that often lack diversity and spontaneity. Although large language models (LLMs) exhibit a remarkable capacity for generating coherent and contextually appropriate utterances, they may fall short when tasked with generating responses that effectively guide users towards their goals, particularly in dialogues with intricate constraints and requirements. This paper introduces DuetSim, a novel framework designed to address the intricate demands of task-oriented dialogues by leveraging LLMs. DuetSim stands apart from conventional approaches by employing two LLMs in tandem: one dedicated to response generation and the other focused on verification. This dual LLM approach empowers DuetSim to produce responses that not only exhibit diversity but also demonstrate accuracy and are preferred by human users. We validate the efficacy of our method through extensive experiments conducted on the MultiWOZ dataset, highlighting improvements in response quality and correctness, largely attributed to the incorporation of the second LLM. Our code is accessible at: https://github.com/suntea233/DuetSim.