# Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk

arXiv: https://arxiv.org/abs/2401.05033 · HTML: https://mlsanity.com/paper/2401.05033/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Learning to Speak and Act in a Fantasy Text Adventure Game](https://mlsanity.com/paper/1903.03094/) (2019)
- [Learning to Write with Cooperative Discriminators](https://mlsanity.com/paper/1805.06087/) (2018)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.00783/) (2021)
- [Building a Conversational Agent Overnight with Dialogue Self-Play](https://mlsanity.com/paper/1801.04871/) (2018)
- [Self-Verification Improves Few-Shot Clinical Information Extraction](https://mlsanity.com/paper/2306.00024/) (2023)
- [Controllable Dialogue Simulation with In-Context Learning](https://mlsanity.com/paper/2210.04185/) (2022)
- [SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Prompting](https://mlsanity.com/paper/2305.09067/) (2023)
- [Disembodied Machine Learning: On the Illusion of Objectivity in NLP](https://mlsanity.com/paper/2101.11974/) (2021)
- [An optimal transportation approach for assessing almost stochastic order](https://mlsanity.com/paper/1705.01788/) (2017)

## Built on by (most cited first, 30 of 34)

- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024)
- [Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement](https://mlsanity.com/paper/2406.11176/) (2024)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [A Survey on the Optimization of Large Language Model-based Agents](https://mlsanity.com/paper/2503.12434/) (2025)
- [Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification](https://mlsanity.com/paper/2406.07515/) (2024)
- [EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding](https://mlsanity.com/paper/2409.01577/) (2024)
- [How to Synthesize Text Data without Model Collapse?](https://mlsanity.com/paper/2412.14689/) (2024)
- [Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation](https://mlsanity.com/paper/2405.13037/) (2024)
- [MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Security](https://mlsanity.com/paper/2508.12538/) (2025)
- [GLEE: A Unified Framework and Benchmark for Language-based Economic Environments](https://mlsanity.com/paper/2410.05254/) (2024)
- [Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games](https://mlsanity.com/paper/2401.17435/) (2024)
- [Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey](https://mlsanity.com/paper/2503.01513/) (2025)
- [Why Less is More (Sometimes): A Theory of Data Curation](https://mlsanity.com/paper/2511.03492/) (2025)
- [Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models](https://mlsanity.com/paper/2404.14772/) (2024)
- [AgentRE: An Agent-Based Framework for Navigating Complex Information Landscapes in Relation Extraction](https://mlsanity.com/paper/2409.01854/) (2024)
- [Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignment](https://mlsanity.com/paper/2410.02768/) (2024)
- [Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversations](https://mlsanity.com/paper/2407.04093/) (2024)
- [Expectation Confirmation Preference Optimization for Multi-Turn Conversational Recommendation Agent](https://mlsanity.com/paper/2506.14302/) (2025)
- [Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuning](https://mlsanity.com/paper/2506.02519/) (2025)
- [TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities](https://mlsanity.com/paper/2603.16759/) (2026)
- [Controllable and Reliable Knowledge-Intensive Task-Oriented Conversational Agents with Declarative Genie Worksheets](https://mlsanity.com/paper/2407.05674/) (2024)
- [What should I wear to a party in a Greek taverna? Evaluation for Conversational Agents in the Fashion Domain](https://mlsanity.com/paper/2408.08907/) (2024)
- [LLMs are Superior Feedback Providers: Bootstrapping Reasoning for Lie Detection with Self-Generated Feedback](https://mlsanity.com/paper/2408.13915/) (2024)
- [Designing Synthetic Discussion Generation Systems: A Case Study for Online Facilitation](https://mlsanity.com/paper/2503.16505/) (2025)
- [LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability](https://mlsanity.com/paper/2607.06157/) (2026)
- [MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders](https://mlsanity.com/paper/2410.06845/) (2024)
- [METEOR: Evolutionary Journey of Large Language Models from Guidance to Self-Growth](https://mlsanity.com/paper/2411.11933/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are powerful dialogue agents, but specializing them towards fulfilling a specific function can be challenging. Instructing tuning, i.e. tuning models on instruction and sample responses generated by humans (Ouyang et al., 2022), has proven as an effective method to do so, yet requires a number of data samples that a) might not be available or b) costly to generate. Furthermore, this cost increases when the goal is to make the LLM follow a specific workflow within a dialogue instead of single instructions. Inspired by the self-play technique in reinforcement learning and the use of LLMs to simulate human agents, we propose a more effective method for data collection through LLMs engaging in a conversation in various roles. This approach generates a training data via "self-talk" of LLMs that can be refined and utilized for supervised fine-tuning. We introduce an automated way to measure the (partial) success of a dialogue. This metric is used to filter the generated conversational data that is fed back in LLM for training. Based on our automated and human evaluations of conversation quality, we demonstrate that such self-talk data improves results. In addition, we examine the various characteristics that showcase the quality of generated dialogues and how they can be connected to their potential utility as training data.