# Implicit Bias in LLMs: A Survey

arXiv: https://arxiv.org/abs/2503.02776 · HTML: https://mlsanity.com/paper/2503.02776/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 38 of its 72 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 38)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- [Fairness in Deep Learning: A Computational Perspective](https://mlsanity.com/paper/1908.08843/) (2019)
- [Detoxifying Large Language Models via Knowledge Editing](https://mlsanity.com/paper/2403.14472/) (2024)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://mlsanity.com/paper/2109.03858/) (2021)
- [PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction](https://mlsanity.com/paper/2010.13816/) (2020)
- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [Dialect prejudice predicts AI decisions about people's character, employability, and criminality](https://mlsanity.com/paper/2403.00742/) (2024)
- [Uncovering Implicit Gender Bias in Narratives through Commonsense Inference](https://mlsanity.com/paper/2109.06437/) (2021)
- [Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation](https://mlsanity.com/paper/2311.00306/) (2023)
- [Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans](https://mlsanity.com/paper/2401.08495/) (2024)
- [Men Are Elected, Women Are Married: Events Gender Bias on Wikipedia](https://mlsanity.com/paper/2106.01601/) (2021)
- [Bias patterns in the application of LLMs for clinical decision support: A comprehensive study](https://mlsanity.com/paper/2404.15149/) (2024)

## Built on by (most cited first, 8 of 8)

- [Bias Association Discovery Framework for Open-Ended LLM Generations](https://mlsanity.com/paper/2508.01412/) (2025)
- [Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairness](https://mlsanity.com/paper/2602.02932/) (2026)
- [Toward Reasonable Parrots: Why Large Language Models Should Argue with Us by Design](https://mlsanity.com/paper/2505.05298/) (2025)
- [Probing Association Biases in LLM Moderation Over-Sensitivity](https://mlsanity.com/paper/2505.23914/) (2025)
- [A word association network methodology for evaluating implicit biases in LLMs compared to humans](https://mlsanity.com/paper/2510.24488/) (2025)
- [Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study](https://mlsanity.com/paper/2603.24125/) (2026)
- [PaperTok: Exploring the Use of Generative AI for Creating Short-form Videos for Research Communication](https://mlsanity.com/paper/2601.18218/) (2026)
- [Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models](https://mlsanity.com/paper/2604.06213/) (2026)

## Abstract (arXiv)

Due to the implement of guardrails by developers, Large language models (LLMs) have demonstrated exceptional performance in explicit bias tests. However, bias in LLMs may occur not only explicitly, but also implicitly, much like humans who consciously strive for impartiality yet still harbor implicit bias. The unconscious and automatic nature of implicit bias makes it particularly challenging to study. This paper provides a comprehensive review of the existing literature on implicit bias in LLMs. We begin by introducing key concepts, theories and methods related to implicit bias in psychology, extending them from humans to LLMs. Drawing on the Implicit Association Test (IAT) and other psychological frameworks, we categorize detection methods into three primary approaches: word association, task-oriented text generation and decision-making. We divide our taxonomy of evaluation metrics for implicit bias into two categories: single-value-based metrics and comparison-value-based metrics. We classify datasets into two types: sentences with masked tokens and complete sentences, incorporating datasets from various domains to reflect the broad application of LLMs. Although research on mitigating implicit bias in LLMs is still limited, we summarize existing efforts and offer insights on future challenges. We aim for this work to serve as a clear guide for researchers and inspire innovative ideas to advance exploration in this task.