# Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

arXiv: https://arxiv.org/abs/2605.20410 · HTML: https://mlsanity.com/paper/2605.20410/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 19 May 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- [Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification](https://mlsanity.com/paper/2504.05419/) (2025)
- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning](https://mlsanity.com/paper/2402.18312/) (2024)
- [Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting](https://mlsanity.com/paper/2401.15585/) (2024)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responses](https://mlsanity.com/paper/2406.05659/) (2024)
- [Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads](https://mlsanity.com/paper/2311.10395/) (2023)
- [Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models](https://mlsanity.com/paper/2410.22517/) (2024)
- [Rethinking Prompt-based Debiasing in Large Language Models](https://mlsanity.com/paper/2503.09219/) (2025)
- [Bias after Prompting: Persistent Discrimination in Large Language Models](https://mlsanity.com/paper/2509.08146/) (2025)
- [Inference-Time Reasoning Selectively Reduces Implicit Social Bias in Large Language Models](https://mlsanity.com/paper/2602.04742/) (2026)
- [Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs](https://mlsanity.com/paper/2509.15020/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model's internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements stem from memorization and familiarity with the dataset rather than genuine understanding of bias.