mlsanityData to 7 Oct 2026

Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs

arXiv 2311.04892 ↗First posted 8 Nov 2023cs.CLcited by 191

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent works have showcased the ability of LLMs to embody diverse personas in their responses, exemplified by prompts like 'You are Yoda. Explain the Theory of Relativity.' While this ability allows personalization of LLMs and enables human behavior simulation, its effect on LLMs' capabilities remains unclear. To fill this gap, we present the first extensive study of the unintended side-effects of persona assignment on the ability of LLMs to perform basic reasoning tasks. Our study covers 24 reasoning datasets, 4 LLMs, and 19 diverse personas (e.g. an Asian person) spanning 5 socio-demographic groups. Our experiments unveil that LLMs harbor deep rooted bias against various socio-demographics underneath a veneer of fairness. While they overtly reject stereotypes when explicitly asked ('Are Black people less skilled at mathematics?'), they manifest stereotypical and erroneous presumptions when asked to answer questions while adopting a persona. These can be observed as abstentions in responses, e.g., 'As a Black person, I can't answer this question as it requires math knowledge', and generally result in a substantial performance drop. Our experiments with ChatGPT-3.5 show that this bias is ubiquitous - 80% of our personas demonstrate bias; it is significant - some datasets show performance drops of 70%+; and can be especially harmful for certain groups - some personas suffer statistically significant drops on 80%+ of the datasets. Overall, all 4 LLMs exhibit this bias to varying extents, with GPT-4-Turbo showing the least but still a problematic amount of bias (evident in 42% of the personas). Further analysis shows that these persona-induced errors can be hard-to-discern and hard-to-avoid. Our findings serve as a cautionary tale that the practice of assigning personas to LLMs - a trend on the rise - can surface their deep-rooted biases and have unforeseeable and detrimental side-effects.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 38 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

26 of 38 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
14 more
  1. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  3. 2018Gender Bias in Coreference Resolutioncited by 438
  4. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  5. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  6. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  7. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  8. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  9. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  10. 2023A Survey on Fairness in Large Language Modelscited by 91
  11. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  12. 2023Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systemscited by 37
  13. 2021Revealing Persona Biases in Dialogue Systemscited by 33
  14. 2023From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 54; 2025: 75; 2026: 60.

191 papers.

Sort:

By area:

  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  2. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  3. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  5. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 89
  6. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  7. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  8. 2024Building Guardrails for Large Language Modelscited by 61
  9. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  10. 2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactionscited by 50
  11. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 48
  12. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
38 more
  1. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  2. 2024Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Contextcited by 41
  3. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  4. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 40
  5. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 39
  6. 2024Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processescited by 38
  7. 2024Personality Alignment of Large Language Modelscited by 36
  8. 2024"You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendationscited by 33
  9. 2024LLMs generate structurally realistic social networks but overestimate political homophilycited by 31
  10. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  11. 2024Persona is a Double-edged Sword: Mitigating the Negative Impact of Role-playing Prompts in Zero-shot Reasoning Taskscited by 28
  12. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  13. 2024'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participantscited by 27
  14. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 26
  15. 2024Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMscited by 25
  16. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  17. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 24
  18. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  19. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  20. 2024Helpful assistant or fruitful facilitator? Investigating how personas affect language model behaviorcited by 21
  21. 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
  22. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  23. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  24. 2025Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamicscited by 19
  25. 2024PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Modelscited by 16
  26. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 16
  27. 2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14
  28. 2024Self-Cognition in Large Language Models: An Exploratory Studycited by 14
  29. 2024GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancycited by 14
  30. 2025Scaffolding Empathy: Training Counselors with Simulated Patients and Utterance-level Performance Visualizationscited by 14
  31. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theorycited by 14
  32. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparencycited by 14
  33. 2026The Need for a Socially-Grounded Persona Framework for User Simulationcited by 14
  34. 2024Do LLMs have Consistent Values?cited by 13
  35. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  36. 2024IMBUE: Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interactioncited by 12
  37. 2024Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Modelscited by 12
  38. 2024Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMscited by 11

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Conditional Cognitive Biases in LLMs: How Biased User Turns Modulate In-Context Reasoning
  2. 2025Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles
  3. 2025Persona-Assigned Large Language Models Exhibit Human-Like Motivated Reasoning
  4. 2024How far can bias go? Tracing bias from pretraining data to alignment
  5. 2025Do Persona-Infused LLMs Affect Performance in a Strategic Reasoning Game?
  6. 2025Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning
  7. 2026Do LLMs Experience an Internal Polylogue? Investigating Reasoning through the Lens of Personas
  8. 2026Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind