# Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions

arXiv: https://arxiv.org/abs/2312.02913 · HTML: https://mlsanity.com/paper/2312.02913/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [BERT with History Answer Embedding for Conversational Question Answering](https://mlsanity.com/paper/1905.05412/) (2019)
- [LLM-powered Data Augmentation for Enhanced Cross-lingual Performance](https://mlsanity.com/paper/2305.14288/) (2023)
- [Evaluating Conversational Recommender Systems via User Simulation](https://mlsanity.com/paper/2006.08732/) (2020)
- [Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems](https://mlsanity.com/paper/2105.03748/) (2021)
- [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023)
- [Look before you Hop: Conversational Question Answering over Knowledge Graphs Using Judicious Context Expansion](https://mlsanity.com/paper/1910.03262/) (2019)
- [FlowQA: Grasping Flow in History for Conversational Machine Comprehension](https://mlsanity.com/paper/1810.06683/) (2018)
- [Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System](https://mlsanity.com/paper/2306.09821/) (2023)
- [Conversational Question Answering: A Survey](https://mlsanity.com/paper/2106.00874/) (2021)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [Evaluating Mixed-initiative Conversational Search Systems via User Simulation](https://mlsanity.com/paper/2204.08046/) (2022)
- [Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond](https://mlsanity.com/paper/2304.13874/) (2023)
- [FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension](https://mlsanity.com/paper/1908.05117/) (2019)
- [RoR: Read-over-Read for Long Document Machine Reading Comprehension](https://mlsanity.com/paper/2109.04780/) (2021)

## Built on by (most cited first, 30 of 70)

- [Faithful Persona-based Conversational Dataset Generation with Large Language Models](https://mlsanity.com/paper/2312.10007/) (2023)
- [Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions](https://mlsanity.com/paper/2408.15787/) (2024)
- [Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?](https://mlsanity.com/paper/2410.23426/) (2024)
- [Examining Identity Drift in Conversations of LLM Agents](https://mlsanity.com/paper/2412.00804/) (2024)
- [A Survey on Recent Advances in Conversational Data Generation](https://mlsanity.com/paper/2405.13003/) (2024)
- [Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents](https://mlsanity.com/paper/2505.10961/) (2025)
- [Self-seeding and Multi-intent Self-instructing LLMs for Generating Intent-aware Information-Seeking dialogs](https://mlsanity.com/paper/2402.11633/) (2024)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants](https://mlsanity.com/paper/2405.02637/) (2024)
- [Generative Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2407.21491/) (2024)
- [Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue](https://mlsanity.com/paper/2409.08330/) (2024)
- [Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets](https://mlsanity.com/paper/2503.09902/) (2025)
- [SynthTRIPs: A Knowledge-Grounded Framework for Benchmark Query Generation for Personalized Tourism Recommenders](https://mlsanity.com/paper/2504.09277/) (2025)
- [EvoEmo: Towards Evolved Emotional Policies for Adversarial LLM Agents in Multi-Turn Price Negotiation](https://mlsanity.com/paper/2509.04310/) (2025)
- [Multi-Document Grounded Multi-Turn Synthetic Dialog Generation](https://mlsanity.com/paper/2409.11500/) (2024)
- [Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring](https://mlsanity.com/paper/2505.08351/) (2025)
- [Conformal Information Pursuit for Interactively Guiding Large Language Models](https://mlsanity.com/paper/2507.03279/) (2025)
- [SteerX: Disentangled Steering for LLM Personalization](https://mlsanity.com/paper/2510.22256/) (2025)
- [MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering](https://mlsanity.com/paper/2404.12926/) (2024)
- [LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering](https://mlsanity.com/paper/2406.06621/) (2024)
- [X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents](https://mlsanity.com/paper/2408.09853/) (2024)
- [Understanding Communication Preferences of Information Workers in Engagement with Text-Based Conversational Agents](https://mlsanity.com/paper/2410.20468/) (2024)
- [INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Models](https://mlsanity.com/paper/2412.11388/) (2024)
- [A New Era in Human Factors Engineering: A Survey of the Applications and Prospects of Large Multimodal Models](https://mlsanity.com/paper/2405.13426/) (2024)
- [A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Engine](https://mlsanity.com/paper/2409.00636/) (2024)
- [Query Understanding in LLM-based Conversational Information Seeking](https://mlsanity.com/paper/2504.06356/) (2025)
- [AgentSense: LLMs Empower Generalizable and Explainable Web-Based Participatory Urban Sensing](https://mlsanity.com/paper/2510.19661/) (2025)
- [Focus Agent: LLM-Powered Virtual Focus Group](https://mlsanity.com/paper/2409.01907/) (2024)
- [A Multimodal Symphony: Integrating Taste and Sound through Generative AI](https://mlsanity.com/paper/2503.02823/) (2025)
- [AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments](https://mlsanity.com/paper/2506.11773/) (2025)

## Abstract (arXiv)

Conversational question-answering (CQA) systems aim to create interactive search systems that effectively retrieve information by interacting with users. To replicate human-to-human conversations, existing work uses human annotators to play the roles of the questioner (student) and the answerer (teacher). Despite its effectiveness, challenges exist as human annotation is time-consuming, inconsistent, and not scalable. To address this issue and investigate the applicability of large language models (LLMs) in CQA simulation, we propose a simulation framework that employs zero-shot learner LLMs for simulating teacher-student interactions. Our framework involves two LLMs interacting on a specific topic, with the first LLM acting as a student, generating questions to explore a given search topic. The second LLM plays the role of a teacher by answering questions and is equipped with additional information, including a text on the given topic. We implement both the student and teacher by zero-shot prompting the GPT-4 model. To assess the effectiveness of LLMs in simulating CQA interactions and understand the disparities between LLM- and human-generated conversations, we evaluate the simulated data from various perspectives. We begin by evaluating the teacher's performance through both automatic and human assessment. Next, we evaluate the performance of the student, analyzing and comparing the disparities between questions generated by the LLM and those generated by humans. Furthermore, we conduct extensive analyses to thoroughly examine the LLM performance by benchmarking state-of-the-art reading comprehension models on both datasets. Our results reveal that the teacher LLM generates lengthier answers that tend to be more accurate and complete. The student LLM generates more diverse questions, covering more aspects of a given topic.