mlsanityData to 7 Oct 2026

Stereotype or Personalization? User Identity Biases Chatbot Recommendations

arXiv 2410.05613 ↗First posted 8 Oct 2024cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

While personalized recommendations are often desired by users, it can be difficult in practice to distinguish cases of bias from cases of personalization: we find that models generate racially stereotypical recommendations regardless of whether the user revealed their identity intentionally through explicit indications or unintentionally through implicit cues. We demonstrate that when people use large language models (LLMs) to generate recommendations, the LLMs produce responses that reflect both what the user wants and who the user is. We argue that chatbots ought to transparently indicate when recommendations are influenced by a user's revealed identity characteristics, but observe that they currently fail to do so. Our experiments show that even though a user's revealed identity significantly influences model recommendations (p < 0.001), model responses obfuscate this fact in response to user queries. This bias and lack of transparency occurs consistently across multiple popular consumer LLMs and for four American racial groups.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 26 of its 68 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

26 of 68 references.

References · 26

  1. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  3. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  4. 2023Explainability for Large Language Models: A Surveycited by 331
  5. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  6. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  7. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  8. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  9. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  10. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  11. 2018Conversational Recommender Systemcited by 115
  12. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 113
14 more
  1. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  2. 2023PALR: Personalization Aware LLMs for Recommendationcited by 96
  3. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  4. 2024Understanding User Experience in Large Language Model Interactionscited by 43
  5. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 43
  6. 2023The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendationscited by 41
  7. 2022CommunityLM: Probing Partisan Worldviews from Language Modelscited by 36
  8. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 35
  9. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  10. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  11. 2019(Un)informed Consent: Studying GDPR Consent Notices in the Fieldcited by 23
  12. 2024Open (Clinical) LLMs are Sensitive to Instruction Phrasingscited by 23
  13. 2024AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filterscited by 21
  14. 2023Adapting LLMs for Efficient, Personalized Information Retrieval: Methods and Implicationscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 16.

33 papers.

Sort:

By area:

  1. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  2. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  3. 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
  4. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 20
  5. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  6. 2025Detecting Prefix Bias in LLM-based Reward Modelscited by 11
  7. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  8. 2025Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American Englishcited by 8
  9. 2025Language Models Change Facts Based on the Way You Talkcited by 8
  10. 2025The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behaviorcited by 8
  11. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoningcited by 5
  12. 2025Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answeringcited by 4
21 more
  1. 2024The Social Impact of Generative LLM-Based AIcited by 3
  2. 2025Societal Impacts Research Requires Benchmarks for Creative Composition Taskscited by 3
  3. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 3
  4. 2026Locating and Controlling Implicit Personalization in Large Language Modelscited by 2
  5. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1
  6. 2025Evaluating Large Language Models for Fair and Reliable Organ Allocationcited by 1
  7. 2025Beyond Text: Characterizing Domain Expert Needs in Document Researchcited by 1
  8. 2025Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientationcited by 1
  9. 2026Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat: A Prominence-Stratified Cross-Provider Auditcited by 1
  10. 2026Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answerscited by 1
  11. 2026Affective Context Amplifies Sycophancy in LLM Responsescited by 1
  12. 2026Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Educationcited by 0
  13. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  14. 2026Playing Games with My Heart: An Evaluation of AI Companion Appscited by 0
  15. 2026VERA-MH: Validation of Ethical and Responsible AI in Mental Healthcited by 0
  16. 2026LLMs Infer Cultural Context but Fail to Apply It When Respondingcited by 0
  17. 2026CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queriescited by 0
  18. 2026Responsible Personalisation: The Double-Edged Sword of Personalisation in Human-Robot Interactioncited by 0
  19. 2026The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleadscited by 0
  20. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0
  21. 2026The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysiscited by 0