mlsanityData to 7 Oct 2026

Automated Survey Collection with LLM-based Conversational Agents

arXiv 2504.02891 ↗First posted 2 Apr 2025cs.CL cs.AIcited by 6

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Objective: Traditional phone-based surveys are among the most accessible and widely used methods to collect biomedical and healthcare data, however, they are often costly, labor intensive, and difficult to scale effectively. To overcome these limitations, we propose an end-to-end survey collection framework driven by conversational Large Language Models (LLMs). Materials and Methods: Our framework consists of a researcher responsible for designing the survey and recruiting participants, a conversational phone agent powered by an LLM that calls participants and administers the survey, a second LLM (GPT-4o) that analyzes the conversation transcripts generated during the surveys, and a database for storing and organizing the results. To test our framework, we recruited 8 participants consisting of 5 native and 3 non-native english speakers and administered 40 surveys. We evaluated the correctness of LLM-generated conversation transcripts, accuracy of survey responses inferred by GPT-4o and overall participant experience. Results: Survey responses were successfully extracted by GPT-4o from conversation transcripts with an average accuracy of 98% despite transcripts exhibiting an average per-line word error rate of 7.7%. While participants noted occasional errors made by the conversational LLM agent, they reported that the agent effectively conveyed the purpose of the survey, demonstrated good comprehension, and maintained an engaging interaction. Conclusions: Our study highlights the potential of LLM agents in conducting and analyzing phone surveys for healthcare applications. By reducing the workload on human interviewers and offering a scalable solution, this approach paves the way for real-world, end-to-end AI-powered phone survey collection systems.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 11 of its 38 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

11 of 38 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  5. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  7. 2024Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Datacited by 65
  8. 2023CXR-LLAVA: a multimodal large language model for interpreting chest X-ray imagescited by 37
  9. 2023LeafAI: query generator for clinical cohort discovery rivaling a human programmercited by 8
  10. 2023WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Modelscited by 7
  11. 2024Extracting Social Determinants of Health from Pediatric Patient Notes Using Large Language Models: Novel Corpus and Methodscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025VoiceAgentEval: A Dual-Dimensional Benchmark for Expert-Level Intelligent Voice-Agent Evaluation of Xbench's Professional-Aligned Seriescited by 4
  3. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  4. 2026INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detectioncited by 1
  5. 2026Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responsescited by 0
  6. 2026Improving Survey Participation in Low-Literacy Populations Through Value-Sensitive Conversational AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Answering real-world clinical questions using large language model based systems
  2. 2023A Large Language Model Approach to Educational Survey Feedback Analysis
  3. 2026On the Credibility of Evaluating LLMs using Survey Questions
  4. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals
  5. 2025Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
  7. 2025In Silico Development of Psychometric Scales: Feasibility of Representative Population Data Simulation with LLMs
  8. 2024Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study