mlsanityData to 7 Oct 2026

Evaluating and Mitigating Discrimination in Language Model Decisions

arXiv 2312.03689 ↗First posted 6 Dec 2023cs.CLcited by 117

Not on the map. Why?

Abstract (arXiv)

As language models (LMs) advance, interest is growing in applying them to high-stakes societal decisions, such as determining financing or housing eligibility. However, their potential for discrimination in such contexts raises ethical concerns, motivating the need for better methods to evaluate these risks. We present a method for proactively evaluating the potential discriminatory impact of LMs in a wide range of use cases, including hypothetical use cases where they have not yet been deployed. Specifically, we use an LM to generate a wide array of potential prompts that decision-makers may input into an LM, spanning 70 diverse decision scenarios across society, and systematically vary the demographic information in each prompt. Applying this methodology reveals patterns of both positive and negative discrimination in the Claude 2.0 model in select settings when no interventions are applied. While we do not endorse or permit the use of language models to make automated decisions for the high-risk use cases we study, we demonstrate techniques to significantly decrease both positive and negative discrimination through careful prompt engineering, providing pathways toward safer deployment in use cases where they may be appropriate. Our work enables developers and policymakers to anticipate, measure, and address discrimination as language model capabilities and applications continue to expand. We release our dataset and prompts at https://huggingface.co/datasets/Anthropic/discrim-eval

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 29 of its 71 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

29 of 71 references.

References · 29

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2022Red Teaming Language Models with Language Modelscited by 983
  4. 2023BloombergGPT: A Large Language Model for Financecited by 923
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  8. 2017Algorithmic decision making and the cost of fairnesscited by 402
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  11. 2018Actionable Recourse in Linear Classificationcited by 327
  12. 2022Prompting GPT-3 To Be Reliablecited by 258
17 more
  1. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  2. 2017Avoiding Discrimination through Causal Reasoningcited by 191
  3. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  4. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
  5. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  6. 2013Discrimination in Online Ad Deliverycited by 93
  7. 2023A Survey on Fairness in Large Language Modelscited by 91
  8. 2023Eliciting Human Preferences with Language Modelscited by 90
  9. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  10. 2022The Fallacy of AI Functionalitycited by 88
  11. 2014Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discriminationcited by 83
  12. 2016Auditing Black-box Models for Indirect Influencecited by 41
  13. 2023Are Emily and Greg Still More Employable than Lakisha and Jamal? Investigating Algorithmic Hiring Bias in the Era of ChatGPTcited by 23
  14. 2021Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studiescited by 22
  15. 2021Problematic Machine Behavior: A Systematic Literature Review of Algorithm Auditscited by 20
  16. 2021Auditing for Discrimination in Algorithms Delivering Job Adscited by 16
  17. 2017Proxy Non-Discrimination in Data-Driven Systemscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 36; 2026: 39.

117 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  6. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  7. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  8. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  9. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69
  10. 2025Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providerscited by 67
  11. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  12. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
38 more
  1. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  2. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 49
  3. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  4. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  5. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
  6. 2024Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?cited by 24
  7. 2024Evaluating language models as risk scorescited by 24
  8. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  9. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  10. 2024The Impossibility of Fair LLMscited by 22
  11. 2024Prompting Fairness: Integrating Causality to Debias Large Language Modelscited by 20
  12. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  13. 2025Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMscited by 20
  14. 2024Auditing the Use of Language Models to Guide Hiring Decisionscited by 19
  15. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  16. 2024Large Language Models Still Exhibit Bias in Long Textcited by 16
  17. 2025Robustly Improving LLM Fairness in Realistic Settings via Interpretabilitycited by 16
  18. 2025Evaluating Bias in LLMs for Job-Resume Matching: Gender, Race, and Educationcited by 13
  19. 2024AGR: Age Group fairness Reward for Bias Mitigation in LLMscited by 12
  20. 2024Acceptable Use Policies for Foundation Modelscited by 12
  21. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  22. 2024Fairness Definitions in Language Models Explainedcited by 10
  23. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  24. 2025AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessmentcited by 10
  25. 2024The Impact of Inference Acceleration on Bias of LLMscited by 9
  26. 2025Guiding LLM Decision-Making with Fairness Reward Modelscited by 9
  27. 2025DAIQ: Auditing Demographic Attribute Inference from Question in LLMscited by 9
  28. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  29. 2024Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Frameworkcited by 8
  30. 2024Towards Compositionality in Concept Learningcited by 7
  31. 2024Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarioscited by 7
  32. 2024SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibrationcited by 6
  33. 2024Improving LLM Group Fairness on Tabular Data via In-Context Learningcited by 6
  34. 2025Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowdscited by 6
  35. 2025Debiasing Reward Models by Representation Learning with Guaranteescited by 6
  36. 2026CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMscited by 6
  37. 2024Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviorscited by 5
  38. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5

50 most cited of 117 All 117 (JSON)