mlsanityData to 7 Oct 2026

'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants

arXiv 2409.19430 ↗First posted 28 Sep 2024cs.HC cs.CL cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

The recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development--e.g., through surveys, experiments, and interviews--with synthetic research data generated by large language models (LLMs). We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift. Initially skeptical, researchers were surprised to see similar narratives emerge in the LLM-generated data when using the interview probe. However, over several conversational turns, they went on to identify fundamental limitations, such as how LLMs foreclose participants' consent and agency, produce responses lacking in palpability and contextual depth, and risk delegitimizing qualitative research methods. We argue that the use of LLMs as proxies for participants enacts the surrogate effect, raising ethical and epistemological concerns that extend beyond the technical limitations of current models to the core of whether LLMs fit within qualitative ways of knowing.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 56 of its 135 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

56 of 135 references.

References · 56

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  10. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
38 more
  1. 2023Whose Opinions Do Language Models Reflect?cited by 759
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  7. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  8. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  10. 2023Language Models can Solve Computer Taskscited by 371
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  12. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  13. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  14. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  15. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  16. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  17. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  18. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  19. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 125
  20. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  21. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  22. 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
  23. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
  24. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  25. 2023Nationality Bias in Text Generationcited by 50
  26. 2023VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototypingcited by 46
  27. 2023AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Predictioncited by 46
  28. 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
  29. 2024AI-Augmented Brainwriting: Investigating the use of LLMs in group ideationcited by 43
  30. 2023CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Modelscited by 34
  31. 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
  32. 2024The illusion of artificial inclusioncited by 34
  33. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
  34. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  35. 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 28
  36. 2024Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosiscited by 23
  37. 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 17
  38. 2022The Data-Production Dispositifcited by 16

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 13.

27 papers.

Sort:

By area:

  1. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
  2. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
  3. 2025A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologiescited by 14
  4. 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
  5. 2025Boosting Relational Deep Learning with Pretrained Tabular Modelscited by 9
  6. 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agentscited by 8
  7. 2025Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactionscited by 4
  8. 2025What Makes LLM Agent Simulations Useful for Policy Practice? An Iterative Design Study in Emergency Preparednesscited by 2
  9. 2026Training Computer Use Agents to Assess the Usability of Graphical User Interfacescited by 2
  10. 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisionscited by 2
  11. 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studiescited by 1
  12. 2025Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulationcited by 1
15 more
  1. 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipationcited by 1
  2. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  3. 2026Interview-Informed Generative Agents for Product Discovery: A Validation Studycited by 1
  4. 2024Collaborative Participatory Research with LLM Agents in South Asia: An Empirically-Grounded Methodological Initiative and Agenda from Field Evidence in Sri Lankacited by 0
  5. 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answeringcited by 0
  6. 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Testscited by 0
  7. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  8. 2026Slurry-as-a-Service: A Modest Proposal on Scalable Pluralistic Alignment for Nutrient Optimizationcited by 0
  9. 2026Co-Refine: AI-Powered Tool Supporting Qualitative Analysiscited by 0
  10. 2026Beyond Compliance: How AI Could Help Creative Writers by Refusing Themcited by 0
  11. 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudescited by 0
  12. 2026Position: Fairness Failure in Generative Models is an Evaluation Problemcited by 0
  13. 2026Runtime Action Interference for AI Control of AlphaStar in StarCraft IIcited by 0
  14. 2026Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLMcited by 0
  15. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring Qualitative Research Using LLMs
  2. 2026Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods
  3. 2024Awes, Laws, and Flaws From Today's LLM Research
  4. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Data
  5. 2025LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis ↗
  6. 2024Reconciling Methodological Paradigms: Employing Large Language Models as Novice Qualitative Research Assistants in Talent Management Research
  7. 2025Recognising, Anticipating, and Mitigating LLM Pollution of Online Behavioural Research
  8. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?