# Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection

arXiv: https://arxiv.org/abs/2501.02295 · HTML: https://mlsanity.com/paper/2501.02295/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 4 Jan 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Quantifying Social Biases Using Templates is Unreliable](https://mlsanity.com/paper/2210.04337/) (2022)
- [Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT](https://mlsanity.com/paper/2407.21058/) (2024)

## Built on by (most cited first, 27 of 27)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Implicit Bias in LLMs: A Survey](https://mlsanity.com/paper/2503.02776/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities](https://mlsanity.com/paper/2508.05525/) (2025)
- [Structured Reasoning for Fairness: A Multi-Agent Approach to Bias Detection in Textual Data](https://mlsanity.com/paper/2503.00355/) (2025)
- [DIF: A Framework for Benchmarking and Verifying Implicit Bias in LLMs](https://mlsanity.com/paper/2505.10013/) (2025)
- [Talent or Luck? Evaluating Attribution Bias in Large Language Models](https://mlsanity.com/paper/2505.22910/) (2025)
- [A word association network methodology for evaluating implicit biases in LLMs compared to humans](https://mlsanity.com/paper/2510.24488/) (2025)
- [Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models](https://mlsanity.com/paper/2510.16340/) (2025)
- [Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles](https://mlsanity.com/paper/2511.06160/) (2025)
- ["The Dentist is an involved parent, the bartender is not": Revealing Implicit Biases in QA with Implicit BBQ](https://mlsanity.com/paper/2512.06732/) (2025)
- [Cross-Language Bias Examination in Large Language Models](https://mlsanity.com/paper/2512.16029/) (2025)
- [Mitigating Social Desirability Bias in Random Silicon Sampling](https://mlsanity.com/paper/2512.22725/) (2025)
- [Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues](https://mlsanity.com/paper/2601.12868/) (2026)
- [Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context](https://mlsanity.com/paper/2603.07792/) (2026)
- [Redirected, Not Removed: Task-Dependent Stereotyping Reveals the Limits of LLM Alignments](https://mlsanity.com/paper/2604.02669/) (2026)
- [HatePrototypes: Interpretable and Transferable Representations for Implicit and Explicit Hate Speech Detection](https://mlsanity.com/paper/2511.06391/) (2025)
- [When Harmless Words Harm: A New Threat to LLM Safety via Conceptual Triggers](https://mlsanity.com/paper/2511.21718/) (2025)
- [The Straight and Narrow: Do LLMs Possess an Internal Moral Path?](https://mlsanity.com/paper/2601.10307/) (2026)
- [Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts](https://mlsanity.com/paper/2602.22070/) (2026)
- [Can LLMs Capture Expert Uncertainty? A Comparative Analysis of Value Alignment in Ethnographic Qualitative Research](https://mlsanity.com/paper/2603.04897/) (2026)
- [ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues](https://mlsanity.com/paper/2604.01925/) (2026)
- [Artificial Intolerance: Stigmatizing Language in Clinical Documentation Skews Large Language Model Decision-Making](https://mlsanity.com/paper/2605.17228/) (2026)
- [To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias](https://mlsanity.com/paper/2606.24596/) (2026)
- [BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs](https://mlsanity.com/paper/2608.14161/) (2026)
- [Noise Out, Bias In: Targeted Bias Injection in Diffusion Language Models via Closed-Loop Activation Steering](https://mlsanity.com/paper/2610.05894/) (2026)
- [Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks](https://mlsanity.com/paper/2610.14464/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have been shown to exhibit various biases and stereotypes in their generated content. While extensive research has investigated biases in LLMs, prior work has predominantly focused on explicit bias, with minimal attention to implicit bias and the relation between these two forms of bias. This paper presents a systematic framework grounded in social psychology theories to investigate and compare explicit and implicit biases in LLMs. We propose a novel self-reflection-based evaluation framework that operates in two phases: first measuring implicit bias through simulated psychological assessment methods, then evaluating explicit bias by prompting LLMs to analyze their own generated content. Through extensive experiments on advanced LLMs across multiple social dimensions, we demonstrate that LLMs exhibit a substantial inconsistency between explicit and implicit biases: while explicit bias manifests as mild stereotypes, implicit bias exhibits strong stereotypes. We further investigate the underlying factors contributing to this explicit-implicit bias inconsistency, examining the effects of training data scale, model size, and alignment techniques. Experimental results indicate that while explicit bias declines with increased training data and model size, implicit bias exhibits a contrasting upward trend. Moreover, contemporary alignment methods effectively suppress explicit bias but show limited efficacy in mitigating implicit bias.