'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants
Not on the map. Why?
Abstract (arXiv)
The recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development--e.g., through surveys, experiments, and interviews--with synthetic research data generated by large language models (LLMs). We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift. Initially skeptical, researchers were surprised to see similar narratives emerge in the LLM-generated data when using the interview probe. However, over several conversational turns, they went on to identify fundamental limitations, such as how LLMs foreclose participants' consent and agency, produce responses lacking in palpability and contextual depth, and risk delegitimizing qualitative research methods. We argue that the use of LLMs as proxies for participants enacts the surrogate effect, raising ethical and epistemological concerns that extend beyond the technical limitations of current models to the core of whether LLMs fit within qualitative ways of knowing.
Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 56 of its 135 references that are in the map. First posted on arXiv 28 Sep 2024.
Builds on
56 of 135 references.
References · 56
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
- 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
38 more
- 2023Whose Opinions Do Language Models Reflect?cited by 759
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
- 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
- 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
- 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
- 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
- 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
- 2023Language Models can Solve Computer Taskscited by 371
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
- 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
- 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
- 2023Can Large Language Models Transform Computational Social Science?cited by 245
- 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
- 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
- 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
- 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
- 2024Simulating Classroom Education with LLM-Empowered Agentscited by 125
- 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
- 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
- 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
- 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
- 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
- 2023Nationality Bias in Text Generationcited by 50
- 2023VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototypingcited by 46
- 2023AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Predictioncited by 46
- 2023ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identificationcited by 45
- 2024AI-Augmented Brainwriting: Investigating the use of LLMs in group ideationcited by 43
- 2023CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Modelscited by 34
- 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
- 2024The illusion of artificial inclusioncited by 34
- 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
- 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
- 2024Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Conceptscited by 28
- 2024Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosiscited by 23
- 2023Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Modelscited by 17
- 2022The Data-Production Dispositifcited by 16
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 13; 2026: 13.
27 papers.
Sort:
By area:
- 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
- 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
- 2025A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologiescited by 14
- 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
- 2025Boosting Relational Deep Learning with Pretrained Tabular Modelscited by 9
- 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agentscited by 8
- 2025Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactionscited by 4
- 2025What Makes LLM Agent Simulations Useful for Policy Practice? An Iterative Design Study in Emergency Preparednesscited by 2
- 2026Training Computer Use Agents to Assess the Usability of Graphical User Interfacescited by 2
- 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisionscited by 2
- 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studiescited by 1
- 2025Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulationcited by 1
15 more
- 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipationcited by 1
- 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
- 2026Interview-Informed Generative Agents for Product Discovery: A Validation Studycited by 1
- 2024Collaborative Participatory Research with LLM Agents in South Asia: An Empirically-Grounded Methodological Initiative and Agenda from Field Evidence in Sri Lankacited by 0
- 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answeringcited by 0
- 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Testscited by 0
- 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
- 2026Slurry-as-a-Service: A Modest Proposal on Scalable Pluralistic Alignment for Nutrient Optimizationcited by 0
- 2026Co-Refine: AI-Powered Tool Supporting Qualitative Analysiscited by 0
- 2026Beyond Compliance: How AI Could Help Creative Writers by Refusing Themcited by 0
- 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudescited by 0
- 2026Position: Fairness Failure in Generative Models is an Evaluation Problemcited by 0
- 2026Runtime Action Interference for AI Control of AlphaStar in StarCraft IIcited by 0
- 2026Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLMcited by 0
- 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0
- 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulation18 Sep 2026
- 2026Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLM4 Sep 2026
- 2026Position: Fairness Failure in Generative Models is an Evaluation Problem17 Aug 2026
- 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions6 Aug 2026
- 2026Runtime Action Interference for AI Control of AlphaStar in StarCraft II5 Aug 2026
- 2026Training Computer Use Agents to Assess the Usability of Graphical User Interfaces28 Apr 2026
- 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes24 Apr 2026
- 2026Co-Refine: AI-Powered Tool Supporting Qualitative Analysis21 Apr 2026
- 2026Beyond Compliance: How AI Could Help Creative Writers by Refusing Them3 Apr 2026
- 2026Interview-Informed Generative Agents for Product Discovery: A Validation Study10 Mar 2026
- 2026Slurry-as-a-Service: A Modest Proposal on Scalable Pluralistic Alignment for Nutrient Optimization2 Mar 2026
- 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making12 Feb 2026
15 more
- 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation11 Feb 2026
- 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipation27 Nov 2025
- 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests25 Oct 2025
- 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answering24 Oct 2025
- 2025Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation14 Oct 2025
- 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agents13 Oct 2025
- 2025What Makes LLM Agent Simulations Useful for Policy Practice? An Iterative Design Study in Emergency Preparedness26 Sep 2025
- 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles29 Jul 2025
- 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies29 May 2025
- 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents13 Apr 2025
- 2025Boosting Relational Deep Learning with Pretrained Tabular Models7 Apr 2025
- 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions24 Feb 2025
- 2025Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions18 Feb 2025
- 2025A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologies14 Feb 2025
- 2024Collaborative Participatory Research with LLM Agents in South Asia: An Empirically-Grounded Methodological Initiative and Agenda from Field Evidence in Sri Lanka13 Nov 2024
- 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
- 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agentscited by 8
- 2025Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactionscited by 4
- 2025What Makes LLM Agent Simulations Useful for Policy Practice? An Iterative Design Study in Emergency Preparednesscited by 2
- 2026Training Computer Use Agents to Assess the Usability of Graphical User Interfacescited by 2
- 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studiescited by 1
- 2026Interview-Informed Generative Agents for Product Discovery: A Validation Studycited by 1
- 2024Collaborative Participatory Research with LLM Agents in South Asia: An Empirically-Grounded Methodological Initiative and Agenda from Field Evidence in Sri Lankacited by 0
- 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudescited by 0
- 2026Training Computer Use Agents to Assess the Usability of Graphical User Interfaces28 Apr 2026
- 2026From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes24 Apr 2026
- 2026Interview-Informed Generative Agents for Product Discovery: A Validation Study10 Mar 2026
- 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agents13 Oct 2025
- 2025What Makes LLM Agent Simulations Useful for Policy Practice? An Iterative Design Study in Emergency Preparedness26 Sep 2025
- 2025Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies29 May 2025
- 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents13 Apr 2025
- 2025Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions18 Feb 2025
- 2024Collaborative Participatory Research with LLM Agents in South Asia: An Empirically-Grounded Methodological Initiative and Agenda from Field Evidence in Sri Lanka13 Nov 2024
- 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
- 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
- 2025A Taxonomy of Linguistic Expressions That Contribute To Anthropomorphism of Language Technologiescited by 14
- 2025Boosting Relational Deep Learning with Pretrained Tabular Modelscited by 9
- 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisionscited by 2
- 2025Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulationcited by 1
- 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipationcited by 1
- 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
- 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answeringcited by 0
- 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Testscited by 0
- 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
- 2026Co-Refine: AI-Powered Tool Supporting Qualitative Analysiscited by 0
12 most cited of 17 All 17 (JSON)
- 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulation18 Sep 2026
- 2026Aplaud: Adaptive Personalized Low-Rank Decomposition for User-Specific LLM4 Sep 2026
- 2026Position: Fairness Failure in Generative Models is an Evaluation Problem17 Aug 2026
- 2026The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions6 Aug 2026
- 2026Runtime Action Interference for AI Control of AlphaStar in StarCraft II5 Aug 2026
- 2026Co-Refine: AI-Powered Tool Supporting Qualitative Analysis21 Apr 2026
- 2026Beyond Compliance: How AI Could Help Creative Writers by Refusing Them3 Apr 2026
- 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making12 Feb 2026
- 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation11 Feb 2026
- 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipation27 Nov 2025
- 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests25 Oct 2025
- 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answering24 Oct 2025
12 newest of 17 All 17 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2023Exploring Qualitative Research Using LLMs
- 2026Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods
- 2024Awes, Laws, and Flaws From Today's LLM Research
- 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Data
- 2025LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis ↗
- 2024Reconciling Methodological Paradigms: Employing Large Language Models as Novice Qualitative Research Assistants in Talent Management Research
- 2025Recognising, Anticipating, and Mitigating LLM Pollution of Online Behavioural Research
- 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?