mlsanityData to 7 Oct 2026

LLM Roleplay: Simulating Human-Chatbot Interaction

arXiv 2407.03974 ↗First posted 4 Jul 2024cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

The development of chatbots requires collecting a large number of human-chatbot dialogues to reflect the breadth of users' sociodemographic backgrounds and conversational goals. However, the resource requirements to conduct the respective user studies can be prohibitively high and often only allow for a narrow analysis of specific dialogue goals and participant demographics. In this paper, we propose LLM Roleplay: a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. LLM Roleplay can be applied to generate dialogues with any type of chatbot and uses large language models (LLMs) to play the role of textually described personas. To validate our method, we collect natural human-chatbot dialogues from different sociodemographic groups and conduct a user study to compare these with our generated dialogues. We evaluate the capabilities of state-of-the-art LLMs in maintaining a conversation during their embodiment of a specific persona and find that our method can simulate human-chatbot dialogues with a high indistinguishability rate.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

28 of 44 references.

References · 28

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
16 more
  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  4. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  5. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  6. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 140
  7. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 128
  8. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  9. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88
  10. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  11. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  12. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  13. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  14. 2021Designing Toxic Content Classification for a Diversity of Perspectivescited by 63
  15. 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35
  16. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  3. 2024Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialoguecited by 15
  4. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  5. 2026ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenderscited by 10
  6. 2025Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoringcited by 6
  7. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  8. 2025ProfiLLM: An LLM-Based Framework for Implicit Profiling of Chatbot Userscited by 3
  9. 2024Inertia in Moral and Value Judgments of Large Language Modelscited by 2
  10. 2025DialogueForge: LLM Simulation of Human-Chatbot Dialoguecited by 2
  11. 2025V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chatcited by 1
  12. 2025PersonaDrift: A Benchmark for Temporal Anomaly Detection in Language-Based Dementia Monitoringcited by 0
3 more
  1. 2026GenAI for Social Work Field Education: Client Simulation with Real-Time Feedbackcited by 0
  2. 2026Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?cited by 0
  3. 2026One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extractioncited by 0