mlsanityData to 14 Oct 2026

Do LLMs exhibit human-like response biases? A case study in survey design

arXiv 2311.04076 ↗First posted 7 Nov 2023cs.CLcited by 180

Not on the map. Why?

Cited by 23 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As large language models (LLMs) become more capable, there is growing excitement about the possibility of using LLMs as proxies for humans in real-world tasks where subjective labels are desired, such as in surveys and opinion polling. One widely-cited barrier to the adoption of LLMs as proxies for humans in subjective tasks is their sensitivity to prompt wording - but interestingly, humans also display sensitivities to instruction changes in the form of response biases. We investigate the extent to which LLMs reflect human response biases, if at all. We look to survey design, where human response biases caused by changes in the wordings of "prompts" have been extensively explored in social psychology literature. Drawing from these works, we design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior, particularly in models that have undergone RLHF. Furthermore, even if a model shows a significant change in the same direction as humans, we find that they are sensitive to perturbations that do not elicit significant changes in humans. These results highlight the pitfalls of using LLMs as human proxies, and underscore the need for finer-grained characterizations of model behavior. Our code, dataset, and collected samples are available at https://github.com/lindiatjuatja/BiasMonkey

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Builds on 42 of its 78 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

42 of 78 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
30 more
  1. 2019How Can We Know What Language Models Know?cited by 959
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2023Whose Opinions Do Language Models Reflect?cited by 773
  7. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  9. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  10. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  11. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  13. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  14. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  15. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  16. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  17. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  18. 2022Language models show human-like content effects on reasoning taskscited by 173
  19. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 163
  20. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 100
  21. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  22. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  23. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  24. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  25. 2023Language Models Trained on Media Diets Can Predict Public Opinioncited by 30
  26. 2022Use-Case-Grounded Simulations for Explanation Evaluationcited by 19
  27. 2016Robsut Wrod Reocginiton via semi-Character Recurrent Neural Networkcited by 18
  28. 2023Are Language Models Worse than Humans at Following Prompts? It's Complicatedcited by 14
  29. 2022Collateral facilitation in humans and language modelscited by 5
  30. 2023Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 44; 2025: 73; 2026: 61.

180 papers.

Sort:

By area:

  1. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  2. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  3. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  4. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  5. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  6. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  7. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  8. 2023Do LLM Agents Exhibit Social Behavior?cited by 53
  9. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
  10. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 52
  11. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 51
  12. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 42
38 more
  1. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 34
  2. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
  3. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  4. 2024Serial Position Effects of Large Language Modelscited by 30
  5. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  6. 2024Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selectioncited by 29
  7. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  8. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  9. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  10. 2023An In-depth Look at Gemini's Language Abilitiescited by 28
  11. 2024'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participantscited by 28
  12. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 28
  13. 2024Evaluating language models as risk scorescited by 27
  14. 2024The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?cited by 26
  15. 2024Open (Clinical) LLMs are Sensitive to Instruction Phrasingscited by 24
  16. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  17. 2025RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patternscited by 24
  18. 2024Cognitive Biases in Large Language Models: A Survey and Mitigation Experimentscited by 20
  19. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  20. 2024Toward LLM-Agent-Based Modeling of Transportation Systems: A Conceptual Frameworkcited by 19
  21. 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
  22. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 18
  23. 2024"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?cited by 17
  24. 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 16
  25. 2024Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Thinkcited by 14
  26. 2024Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Studycited by 14
  27. 2025Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responsescited by 13
  28. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  29. 2024Mitigating Selection Bias with Node Pruning and Auxiliary Optionscited by 12
  30. 2025From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMscited by 12
  31. 2025LLM-Based Social Simulations Require a Boundarycited by 12
  32. 2024Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMscited by 11
  33. 2025AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessmentcited by 11
  34. 2024Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Modelscited by 10
  35. 2025How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?cited by 10
  36. 2025SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulationcited by 10
  37. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applicationscited by 10
  38. 2025When Two LLMs Debate, Both Think They'll Wincited by 10

50 most cited of 180 All 180 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating LLMs as Human Surrogates in Controlled Experiments
  2. 2025Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs
  3. 2026LLMs Can Better Capture Human Judgments--With the Right Prompts
  4. 2025Evaluating the Bias in LLMs for Surveying Opinion and Decision Making in Healthcare
  5. 2024How far can bias go? Tracing bias from pretraining data to alignment
  6. 2025LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns
  7. 2024Dissecting Human and LLM Preferences
  8. 2026On the Credibility of Evaluating LLMs using Survey Questions