# Synthetic Dialogue Dataset Generation using LLM Agents

arXiv: https://arxiv.org/abs/2401.17461 · HTML: https://mlsanity.com/paper/2401.17461/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 45 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 7 of 7)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions](https://mlsanity.com/paper/2303.08233/) (2023)
- [Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptions](https://mlsanity.com/paper/2209.15565/) (2022)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations](https://mlsanity.com/paper/2004.14530/) (2020)

## Built on by (most cited first, 30 of 39)

- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Proactive Assistant Dialogue Generation from Streaming Egocentric Videos](https://mlsanity.com/paper/2506.05904/) (2025)
- [Enhancing Critical Thinking in Education by means of a Socratic Chatbot](https://mlsanity.com/paper/2409.05511/) (2024)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm](https://mlsanity.com/paper/2502.19103/) (2025)
- [MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification](https://mlsanity.com/paper/2412.04494/) (2024)
- [Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2402.11142/) (2024)
- [A Survey of Data Synthesis Approaches](https://mlsanity.com/paper/2407.03672/) (2024)
- [AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation](https://mlsanity.com/paper/2510.19361/) (2025)
- [X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents](https://mlsanity.com/paper/2408.09853/) (2024)
- [DatasetAgent: A Novel Multi-Agent System for Auto-Constructing Datasets from Real-World Images](https://mlsanity.com/paper/2507.08648/) (2025)
- [Benchmark Everything Everywhere All at Once](https://mlsanity.com/paper/2606.06462/) (2026)
- [Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis](https://mlsanity.com/paper/2404.19622/) (2024)
- [A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Engine](https://mlsanity.com/paper/2409.00636/) (2024)
- [Measuring Diversity in Synthetic Datasets](https://mlsanity.com/paper/2502.08512/) (2025)
- [Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History](https://mlsanity.com/paper/2505.21362/) (2025)
- [We Argue to Agree: Towards Personality-Driven Argumentation-Based Negotiation Dialogue Systems for Tourism](https://mlsanity.com/paper/2509.11118/) (2025)
- [ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysis](https://mlsanity.com/paper/2412.14809/) (2024)
- [AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Difference](https://mlsanity.com/paper/2505.13531/) (2025)
- [BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions](https://mlsanity.com/paper/2506.05766/) (2025)
- [DeepThink: Aligning Language Models with Domain-Specific User Intents](https://mlsanity.com/paper/2502.05497/) (2025)
- [ConvoGen: Enhancing Conversational AI with Synthetic Data: A Multi-Agent Approach](https://mlsanity.com/paper/2503.17460/) (2025)
- [Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questions](https://mlsanity.com/paper/2504.15918/) (2025)
- [DiscussLLM: Teaching Large Language Models When to Speak](https://mlsanity.com/paper/2508.18167/) (2025)
- [HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation](https://mlsanity.com/paper/2606.01230/) (2026)
- [Simulating User Diversity in Task-Oriented Dialogue Systems using Large Language Models](https://mlsanity.com/paper/2502.12813/) (2025)
- [SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection](https://mlsanity.com/paper/2503.15044/) (2025)
- [Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation](https://mlsanity.com/paper/2508.18210/) (2025)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)
- [Task-oriented Prompt Enhancement via Script Generation](https://mlsanity.com/paper/2409.16418/) (2024)

## Abstract (arXiv)

Linear programming (LP) problems are pervasive in real-life applications. However, despite their apparent simplicity, an untrained user may find it difficult to determine the linear model of their specific problem. We envisage the creation of a goal-oriented conversational agent that will engage in conversation with the user to elicit all information required so that a subsequent agent can generate the linear model. In this paper, we present an approach for the generation of sample dialogues that can be used to develop and train such a conversational agent. Using prompt engineering, we develop two agents that "talk" to each other, one acting as the conversational agent, and the other acting as the user. Using a set of text descriptions of linear problems from NL4Opt available to the user only, the agent and the user engage in conversation until the agent has retrieved all key information from the original problem description. We also propose an extrinsic evaluation of the dialogues by assessing how well the summaries generated by the dialogues match the original problem descriptions. We conduct human and automatic evaluations, including an evaluation approach that uses GPT-4 to mimic the human evaluation metrics. The evaluation results show an overall good quality of the dialogues, though research is still needed to improve the quality of the GPT-4 evaluation metrics. The resulting dialogues, including the human annotations of a subset, are available to the research community. The conversational agent used for the generation of the dialogues can be used as a baseline.