# In-Context Learning User Simulators for Task-Oriented Dialog Systems

arXiv: https://arxiv.org/abs/2306.00774 · HTML: https://mlsanity.com/paper/2306.00774/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge](https://mlsanity.com/paper/1901.07931/) (2019)
- [Recent Advances and Challenges in Task-oriented Dialog System](https://mlsanity.com/paper/2003.07490/) (2020)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems](https://mlsanity.com/paper/2002.04793/) (2020)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems](https://mlsanity.com/paper/1805.06966/) (2018)
- [Controllable Dialogue Simulation with In-Context Learning](https://mlsanity.com/paper/2210.04185/) (2022)
- [Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2008.06239/) (2020)
- [A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems](https://mlsanity.com/paper/1607.00070/) (2016)
- [User Modeling for Task Oriented Dialogues](https://mlsanity.com/paper/1811.04369/) (2018)
- [Evaluating Mixed-initiative Conversational Search Systems via User Simulation](https://mlsanity.com/paper/2204.08046/) (2022)
- [Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems](https://mlsanity.com/paper/2106.08838/) (2021)
- [GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers](https://mlsanity.com/paper/2208.10817/) (2022)
- [ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format](https://mlsanity.com/paper/2211.17148/) (2022)
- [Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2203.01552/) (2022)
- [An In-depth Investigation of User Response Simulation for Conversational Search](https://mlsanity.com/paper/2304.07944/) (2023)
- [Few Shot Dialogue State Tracking using Meta-learning](https://mlsanity.com/paper/2101.06779/) (2021)
- [Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator](https://mlsanity.com/paper/2210.14529/) (2022)
- [A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems](https://mlsanity.com/paper/2210.08692/) (2022)

## Built on by (most cited first, 19 of 19)

- [Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2402.13374/) (2024)
- [DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues](https://mlsanity.com/paper/2405.13028/) (2024)
- [A Survey on Recent Advances in Conversational Data Generation](https://mlsanity.com/paper/2405.13003/) (2024)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems](https://mlsanity.com/paper/2404.06762/) (2024)
- [ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders](https://mlsanity.com/paper/2602.16938/) (2026)
- [ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questions](https://mlsanity.com/paper/2407.00942/) (2024)
- [Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants](https://mlsanity.com/paper/2605.09808/) (2026)
- [Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors](https://mlsanity.com/paper/2404.03304/) (2024)
- [Exploring the Impact of Personality Traits on Conversational Recommender Systems: A Simulation with Large Language Models](https://mlsanity.com/paper/2504.12313/) (2025)
- [Game Development as Human-LLM Interaction](https://mlsanity.com/paper/2408.09386/) (2024)
- [From Intents to Conversations: Generating Intent-Driven Dialogues with Contrastive Learning for Multi-Turn Classification](https://mlsanity.com/paper/2411.14252/) (2024)
- [VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity](https://mlsanity.com/paper/2501.03139/) (2025)
- [Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback](https://mlsanity.com/paper/2502.13298/) (2025)
- [FloCA: Towards Faithful and Logically Consistent Flowchart Reasoning](https://mlsanity.com/paper/2602.14035/) (2026)
- [Generating Diverse Personas for User Simulators to Test Interview Dialogue Systems](https://mlsanity.com/paper/2608.19549/) (2026)
- ["Act Like a 5th Grader" is Not Enough: Bounding Knowledge in LLM-Based User Simulators](https://mlsanity.com/paper/2608.30033/) (2026)
- [Pretrained Persona Mixture Models and Tandem Models for Human Simulation](https://mlsanity.com/paper/2609.22607/) (2026)
- [UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions](https://mlsanity.com/paper/2610.11638/) (2026)

## Abstract (arXiv)

This paper presents a novel application of large language models in user simulation for task-oriented dialog systems, specifically focusing on an in-context learning approach. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples. Unlike traditional simulators, this method eliminates the need for labor-intensive rule definition or extensive annotated data, making it more efficient and accessible. Additionally, an error analysis of the interaction between the user simulator and dialog system uncovers common mistakes, providing valuable insights into areas that require improvement. Our implementation is available at https://github.com/telepathylabsai/prompt-based-user-simulator.