mlsanityData to 7 Oct 2026

Synthetic Dialogue Dataset Generation using LLM Agents

arXiv 2401.17461 ↗First posted 30 Jan 2024cs.CL cs.AIcited by 39

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Linear programming (LP) problems are pervasive in real-life applications. However, despite their apparent simplicity, an untrained user may find it difficult to determine the linear model of their specific problem. We envisage the creation of a goal-oriented conversational agent that will engage in conversation with the user to elicit all information required so that a subsequent agent can generate the linear model. In this paper, we present an approach for the generation of sample dialogues that can be used to develop and train such a conversational agent. Using prompt engineering, we develop two agents that "talk" to each other, one acting as the conversational agent, and the other acting as the user. Using a set of text descriptions of linear problems from NL4Opt available to the user only, the agent and the user engage in conversation until the agent has retrieved all key information from the original problem description. We also propose an extrinsic evaluation of the dialogues by assessing how well the summaries generated by the dialogues match the original problem descriptions. We conduct human and automatic evaluations, including an evaluation approach that uses GPT-4 to mimic the human evaluation metrics. The evaluation results show an overall good quality of the dialogues, though research is still needed to improve the quality of the GPT-4 evaluation metrics. The resulting dialogues, including the human annotations of a subset, are available to the research community. The conversational agent used for the generation of the dialogues can be used as a baseline.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 7 of its 45 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

7 of 45 references.

References · 7

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2023NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptionscited by 133
  5. 2022Augmenting Operations Research with Auto-Formulation of Optimization Models from Problem Descriptionscited by 54
  6. 2021Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledgecited by 40
  7. 2020Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversationscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 20; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  2. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  3. 2024Enhancing Critical Thinking in Education by means of a Socratic Chatbotcited by 21
  4. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  5. 2025LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigmcited by 14
  6. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  7. 2024Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extractioncited by 9
  8. 2024A Survey of Data Synthesis Approachescited by 8
  9. 2025AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generationcited by 7
  10. 2024X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agentscited by 6
  11. 2025DatasetAgent: A Novel Multi-Agent System for Auto-Constructing Datasets from Real-World Imagescited by 6
  12. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
27 more
  1. 2024A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Enginecited by 5
  2. 2025Measuring Diversity in Synthetic Datasetscited by 5
  3. 2025We Argue to Agree: Towards Personality-Driven Argumentation-Based Negotiation Dialogue Systems for Tourismcited by 5
  4. 2026Benchmark Everything Everywhere All at Oncecited by 5
  5. 2025AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Differencecited by 4
  6. 2025Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue Historycited by 4
  7. 2025BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactionscited by 4
  8. 2024ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysiscited by 3
  9. 2025DeepThink: Aligning Language Models with Domain-Specific User Intentscited by 2
  10. 2025ConvoGen: Enhancing Conversational AI with Synthetic Data: A Multi-Agent Approachcited by 2
  11. 2025Ask2Loc: Learning to Locate Instructional Visual Answers by Asking Questionscited by 2
  12. 2026HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulationcited by 2
  13. 2025Simulating User Diversity in Task-Oriented Dialogue Systems using Large Language Modelscited by 1
  14. 2025SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detectioncited by 1
  15. 2025DiscussLLM: Teaching Large Language Models When to Speakcited by 1
  16. 2025Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generationcited by 1
  17. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
  18. 2024Task-oriented Prompt Enhancement via Script Generationcited by 0
  19. 2024MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Taskscited by 0
  20. 2025Improving Automated Secure Code Reviews: A Synthetic Dataset for Code Vulnerability Flawscited by 0
  21. 2025Delving into Multilingual Ethical Bias: The MSQAD with Statistical Hypothesis Tests for Large Language Modelscited by 0
  22. 2025DRIVE: Disfluency-Rich Synthetic Dialog Data Generation Framework for Intelligent Vehicle Environmentscited by 0
  23. 2025ProKG-Dial: Progressive Multi-Turn Dialogue Construction with Domain Knowledge Graphscited by 0
  24. 2026UniD3: A Knowledge Graph-Enhanced RAG Framework for Drug-Disease Discovery and Reasoningcited by 0
  25. 2026Translating the Untranslatable: An Operationalizable Ontology for Untranslatabilitycited by 0
  26. 2026Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Modelscited by 0
  27. 2026Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation
  2. 2026GenesisFunc: Multi-Agent Data Generation for Accurate and Generalizable Function-Calling
  3. 2026State-Grounded Multi-Agent Synthetic Data Generation for Tool-Augmented LLMs
  4. 2025Human vs. Agent in Task-Oriented Conversations
  5. 2023A New Dialogue Response Generation Agent for Large Language Models by Asking Questions to Detect User's Intentions
  6. 2024Self-Directed Synthetic Dialogues and Revisions Technical Report
  7. 2025ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction
  8. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations