mlsanityData to 7 Oct 2026

Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems

arXiv 2310.05280 ↗First posted 8 Oct 2023cs.CL cs.AIcited by 37

Not on the map. Why?

Abstract (arXiv)

Recent advancements in Large Language Models empower them to follow freeform instructions, including imitating generic or specific demographic personas in conversations. We define generic personas to represent demographic groups, such as "an Asian person", whereas specific personas may take the form of specific popular Asian names like "Yumi". While the adoption of personas enriches user experiences by making dialogue systems more engaging and approachable, it also casts a shadow of potential risk by exacerbating social biases within model responses, thereby causing societal harm through interactions with users. In this paper, we systematically study "persona biases", which we define to be the sensitivity of dialogue models' harmful behaviors contingent upon the personas they adopt. We categorize persona biases into biases in harmful expression and harmful agreement, and establish a comprehensive evaluation framework to measure persona biases in five aspects: Offensiveness, Toxic Continuation, Regard, Stereotype Agreement, and Toxic Agreement. Additionally, we propose to investigate persona biases by experimenting with UNIVERSALPERSONA, a systematically constructed persona dataset encompassing various types of both generic and specific model personas. Through benchmarking on four different models -- including Blender, ChatGPT, Alpaca, and Vicuna -- our study uncovers significant persona biases in dialogue systems. Our findings also underscore the pressing need to revisit the use of personas in dialogue agents to ensure safe application.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 61 of its 102 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

61 of 102 references.

References · 61

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  11. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
38 more
  1. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  2. 2021GPT Understands, Toocited by 715
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2020Recipes for building an open-domain chatbotcited by 593
  6. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  7. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  9. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  10. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  11. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  12. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  13. 2016A Persona-Based Neural Conversation Modelcited by 221
  14. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  15. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  16. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  17. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  18. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  19. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  20. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  21. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  22. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  23. 2017ParlAI: A Dialog Research Software Platformcited by 139
  24. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  25. 2018Training Millions of Personalized Dialogue Agentscited by 126
  26. 2020Towards Controllable Biases in Language Generationcited by 112
  27. 2017Controllable Abstractive Summarizationcited by 110
  28. 2019Automatically Neutralizing Subjective Bias in Textcited by 105
  29. 2019Does Gender Matter? Towards Fairness in Dialogue Systemscited by 89
  30. 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
  31. 2021On Measures of Biases and Harms in NLPcited by 81
  32. 2021BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Datacited by 75
  33. 2019A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Datacited by 74
  34. 2022TweetNLP: Cutting-Edge Natural Language Processing for Social Mediacited by 71
  35. 2019Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniquescited by 65
  36. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  37. 2018Polite Dialogue Generation Without Parallel Datacited by 57
  38. 2019Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Functioncited by 57

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 17; 2026: 5.

37 papers.

Sort:

By area:

  1. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  2. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  3. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  4. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 39
  5. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  6. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 26
  7. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  8. 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
  9. 2024Helpful assistant or fruitful facilitator? Investigating how personas affect language model behaviorcited by 21
  10. 2024An Empirical Analysis on Large Language Models in Debate Evaluationcited by 20
  11. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
  12. 2025Diffusion Models Through a Global Lens: Are They Culturally Inclusive?cited by 14
25 more
  1. 2025Exploring the Impact of Personality Traits on LLM Bias and Toxicitycited by 14
  2. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparencycited by 14
  3. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
  4. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  5. 2025Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generationcited by 8
  6. 2025From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactionscited by 7
  7. 2024MisinfoEval: Generative AI in the Era of "Alternative Facts"cited by 6
  8. 2025Language Models Predict Empathy Gaps Between Social In-groups and Out-groupscited by 6
  9. 2024Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspectivecited by 5
  10. 2024ERABAL: Enhancing Role-Playing Agents through Boundary-Aware Learningcited by 5
  11. 2025Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Taskscited by 4
  12. 2025AI Will Always Love You: Studying Implicit Biases in Romantic AI Companionscited by 3
  13. 2025EtiCor++: Towards Understanding Etiquettical Bias in LLMscited by 3
  14. 2025User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMscited by 2
  15. 2024White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMscited by 1
  16. 2024UPCS: Unbiased Persona Construction for Dialogue Generationcited by 1
  17. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1
  18. 2025Exploring the Impact of Occupational Personas on Domain-Specific QAcited by 1
  19. 2025IndiCASA: A Dataset and Bias Evaluation Framework in LLMs Using Contrastive Embedding Similarity in the Indian Contextcited by 1
  20. 2026How Agents Represent Humans: Human-Directed Stereotypes in an Open Agent Social Networkcited by 1
  21. 2025Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Researchcited by 0
  22. 2026To Generate or Discriminate? Methodological Considerations for Measuring Cultural Alignment in LLMscited by 0
  23. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  24. 2026Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensionscited by 0
  25. 2026Population Fidelity: Evaluating Population Representativeness in LLMscited by 0