mlsanityData to 14 Oct 2026

Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

arXiv 2305.18189 ↗First posted 29 May 2023cs.CL cs.AI cs.CYcited by 207

Not on the map. Why?

Abstract (arXiv)

To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs. Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexicon or data labeling. Grounded in the sociolinguistic concept of markedness (which characterizes explicitly linguistically marked categories versus unmarked defaults), our proposed method is twofold: 1) prompting an LLM to generate personas, i.e., natural language descriptions, of the target demographic group alongside personas of unmarked, default groups; 2) identifying the words that significantly distinguish personas of the target group from corresponding unmarked ones. We find that the portrayals generated by GPT-3.5 and GPT-4 contain higher rates of racial stereotypes than human-written portrayals using the same prompts. The words distinguishing personas of marked (non-white, non-male) groups reflect patterns of othering and exoticizing these demographics. An intersectional lens further reveals tropes that dominate portrayals of marginalized groups, such as tropicalism and the hypersexualization of minoritized women. These representational harms have concerning implications for downstream applications like story generation.

Key facts. Cited by 207 papers in the map (data through 14 Oct 2026). Builds on 31 of its 128 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

31 of 128 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  6. 2021Ethical and social risks of harm from Language Modelscited by 966
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  8. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  9. 2023Whose Opinions Do Language Models Reflect?cited by 773
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  11. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  12. 2019On Measuring Social Biases in Sentence Encoderscited by 398
19 more
  1. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  2. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  3. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  4. 2020Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biasescited by 167
  5. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  6. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  7. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  8. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  9. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 135
  10. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 108
  11. 2020Multi-Dimensional Gender Bias Classificationcited by 80
  12. 2022American == White in Multimodal Language-and-Image AIcited by 39
  13. 2022Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Modelscited by 35
  14. 2022Evidence for Hypodescent in Visual Semantic AIcited by 30
  15. 2022Markedness in Visual Semantic AIcited by 29
  16. 2022SODAPOP: Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Modelscited by 22
  17. 2020Generalized Word Shift Graphs: A Method for Visualizing and Explaining Pairwise Comparisons Between Textscited by 21
  18. 2020Unequal Representations: Analyzing Intersectional Biases in Word Embeddings Using Representational Similarity Analysiscited by 12
  19. 2021When the Echo Chamber Shatters: Examining the Use of Community-Specific Language Post-Subreddit Bancited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 23; 2024: 63; 2025: 71; 2026: 50.

207 papers.

Sort:

By area:

  1. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  2. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  4. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 199
  5. 2023Simulating Opinion Dynamics with Networks of LLM-based Agentscited by 154
  6. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  7. 2023A Survey on Fairness in Large Language Modelscited by 95
  8. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  9. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  10. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  11. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  12. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
38 more
  1. 2023Rehearsal: Simulating Conflict to Teach Conflict Resolutioncited by 55
  2. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  3. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  4. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  5. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 54
  6. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 51
  7. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 49
  8. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  9. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  10. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  11. 2023ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferencescited by 43
  12. 2023Opportunities and Risks of LLMs for Scalable Deliberation with Poliscited by 42
  13. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 42
  14. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 40
  15. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  16. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 35
  17. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  18. 2024Aligning Language Models with Demonstrated Feedbackcited by 34
  19. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 34
  20. 2024LLMs generate structurally realistic social networks but overestimate political homophilycited by 33
  21. 2024Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humanscited by 31
  22. 2025Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Modelscited by 30
  23. 2023Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implicationscited by 29
  24. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 29
  25. 2025Value Profiles for Encoding Human Variationcited by 29
  26. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  27. 2024AnthroScore: A Computational Linguistic Measure of Anthropomorphismcited by 27
  28. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 27
  29. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 27
  30. 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflectioncited by 27
  31. 2024Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publicationscited by 26
  32. 2024Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?cited by 25
  33. 2025Multilingual Prompting for Improving LLM Generation Diversitycited by 25
  34. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  35. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  36. 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
  37. 2023A Material Lens on Coloniality in NLPcited by 21
  38. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21

50 most cited of 207 All 207 (JSON)