# Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data

arXiv: https://arxiv.org/abs/2301.05843 · HTML: https://mlsanity.com/paper/2301.05843/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 124 references that are in the map.
- First posted on arXiv 14 Jan 2023.

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Zero-Shot Dialogue State Tracking via Cross-Task Transfer](https://mlsanity.com/paper/2109.04655/) (2021)
- [A Taxonomy of Empathetic Response Intents in Human Social Conversations](https://mlsanity.com/paper/2012.04080/) (2020)
- [Effects of Persuasive Dialogues: Testing Bot Identities and Inquiry Strategies](https://mlsanity.com/paper/2001.04564/) (2020)
- [Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models](https://mlsanity.com/paper/2205.00176/) (2022)
- [If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills](https://mlsanity.com/paper/2002.01862/) (2020)
- [LAD: Language Models as Data for Zero-Shot Dialog](https://mlsanity.com/paper/2207.14393/) (2022)
- [Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions](https://mlsanity.com/paper/1905.10700/) (2019)

## Built on by (most cited first, 30 of 48)

- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache](https://mlsanity.com/paper/2401.02669/) (2024)
- [Training-Free Long-Context Scaling of Large Language Models](https://mlsanity.com/paper/2402.17463/) (2024)
- [MindfulDiary: Harnessing Large Language Model to Support Psychiatric Patients' Journaling](https://mlsanity.com/paper/2310.05231/) (2023)
- [Prompt Stealing Attacks Against Large Language Models](https://mlsanity.com/paper/2402.12959/) (2024)
- [LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation](https://mlsanity.com/paper/2305.13614/) (2023)
- [ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events](https://mlsanity.com/paper/2309.12244/) (2023)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [MindShift: Leveraging Large Language Models for Mental-States-Based Problematic Smartphone Use Intervention](https://mlsanity.com/paper/2309.16639/) (2023)
- [Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention](https://mlsanity.com/paper/2402.11353/) (2024)
- [Large Language Model for Mental Health: A Systematic Review](https://mlsanity.com/paper/2403.15401/) (2024)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelines](https://mlsanity.com/paper/2407.12797/) (2024)
- [Evaluation of an LLM in Identifying Logical Fallacies: A Call for Rigor When Adopting LLMs in HCI Research](https://mlsanity.com/paper/2404.05213/) (2024)
- [An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner's Guide](https://mlsanity.com/paper/2402.14837/) (2024)
- [PlanFitting: Personalized Exercise Planning with Large Language Model-driven Conversational Agent](https://mlsanity.com/paper/2309.12555/) (2023)
- [Ain't Misbehavin' -- Using LLMs to Generate Expressive Robot Behavior in Conversations with the Tabletop Robot Haru](https://mlsanity.com/paper/2402.11571/) (2024)
- [ExploreSelf: Fostering User-driven Exploration and Reflection on Personal Challenges with Adaptive Guidance by Large Language Models](https://mlsanity.com/paper/2409.09662/) (2024)
- [Advancing Conversational Psychotherapy: Integrating Privacy, Dual-Memory, and Domain Expertise with Large Language Models](https://mlsanity.com/paper/2412.02987/) (2024)
- [Telephone Surveys Meet Conversational AI: Evaluating a LLM-Based Telephone Survey System at Scale](https://mlsanity.com/paper/2502.20140/) (2025)
- [Exploring Conversational Agents as an Effective Tool for Measuring Cognitive Biases in Decision-Making](https://mlsanity.com/paper/2401.06686/) (2024)
- [Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needs](https://mlsanity.com/paper/2411.19554/) (2024)
- [AI-Enabled Conversational Journaling for Advancing Parkinson's Disease Symptom Tracking](https://mlsanity.com/paper/2503.03532/) (2025)
- [DiaCBT: A Long-Periodic Dialogue Corpus Guided by Cognitive Conceptualization Diagram for CBT-based Psychological Counseling](https://mlsanity.com/paper/2509.02999/) (2025)
- [ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference](https://mlsanity.com/paper/2509.14537/) (2025)
- [Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models](https://mlsanity.com/paper/2406.07288/) (2024)
- [A Systematic Review of Generative AI for Teaching and Learning Practice](https://mlsanity.com/paper/2406.09520/) (2024)
- [Pet-Bench: Benchmarking the Abilities of Large Language Models as E-Pets in Social Network Services](https://mlsanity.com/paper/2506.03761/) (2025)

## Abstract (arXiv)

Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal, such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.