mlsanityData to 7 Oct 2026

Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

arXiv 2409.08330 ↗First posted 12 Sep 2024cs.CL cs.CY cs.HCcited by 15

Not on the map. Why?

Abstract (arXiv)

Studying and building datasets for dialogue tasks is both expensive and time-consuming due to the need to recruit, train, and collect data from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, to what extent do LLM-based simulations textitactually reflect human dialogues? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, demonstrating a systematic divergence along the multiple textual properties, including style and content. Further, in comparisons of English, Chinese, and Russian dialogues, we find that models perform similarly. Our results suggest that LLMs generally perform better when the human themself writes in a way that is more similar to the LLM's own style.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

39 of 63 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  10. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
27 more
  1. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  2. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  3. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  4. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  5. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  6. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  7. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  8. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  9. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  10. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  11. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
  12. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 61
  13. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  14. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 57
  15. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  16. 2023Do LLM Agents Exhibit Social Behavior?cited by 53
  17. 2024Y Social: an LLM-powered Social Media Digital Twincited by 43
  18. 2023Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisionscited by 41
  19. 2024A Survey on Large Language Model Hallucination via a Creativity Perspectivecited by 38
  20. 2022TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languagescited by 21
  21. 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
  22. 2024LLM Roleplay: Simulating Human-Chatbot Interactioncited by 15
  23. 2023Can Language Model Moderators Improve the Health of Online Discourse?cited by 12
  24. 2022Detecting Text Formality: A Study of Text Classification Approachescited by 11
  25. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  26. 2024Role-Play Zero-Shot Prompting with Large Language Models for Open-Domain Human-Machine Conversationcited by 7
  27. 2023PIPPA: A Partially Synthetic Conversational Datasetcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  2. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  3. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  4. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  5. 2025Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoralcited by 14
  6. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
  7. 2026Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviorscited by 4
  8. 2025Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulationcited by 3
  9. 2025Show or Tell? Modeling the evolution of request-making in Human-LLM conversationscited by 3
  10. 2025The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrumcited by 3
  11. 2026Synthetic Users, Real Differences: an Evaluation Framework for User Simulation in Multi-Turn Conversationscited by 2
  12. 2026Re-Centering Humans in LLM Personalizationcited by 0
3 more
  1. 2026Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Promptscited by 0
  2. 2026Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulationscited by 0
  3. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0