# Do Biased Models Have Biased Thoughts?

arXiv: https://arxiv.org/abs/2508.06671 · HTML: https://mlsanity.com/paper/2508.06671/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 17 of its 68 references that are in the map.
- First posted on arXiv 8 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024)
- [Improve LLM-as-a-Judge Ability as a General Ability](https://mlsanity.com/paper/2502.11689/) (2025)
- [Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models](https://mlsanity.com/paper/2408.03907/) (2024)
- [Dissociation of Faithful and Unfaithful Reasoning in LLMs](https://mlsanity.com/paper/2405.15092/) (2024)
- [First-Person Fairness in Chatbots](https://mlsanity.com/paper/2410.19803/) (2024)
- [Chain of Logic: Rule-Based Reasoning with Large Language Models](https://mlsanity.com/paper/2402.10400/) (2024)
- [Should We Attend More or Less? Modulating Attention for Fairness](https://mlsanity.com/paper/2305.13088/) (2023)
- [Data Generation Using Large Language Models for Text Classification: An Empirical Case Study](https://mlsanity.com/paper/2407.12813/) (2024)
- [Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models](https://mlsanity.com/paper/2405.14555/) (2024)

## Built on by (most cited first, 1 of 1)

- [Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking](https://mlsanity.com/paper/2608.17270/) (2026)

## Abstract (arXiv)

The impressive performance of language models is undeniable. However, the presence of biases based on gender, race, socio-economic status, physical appearance, and sexual orientation makes the deployment of language models challenging. This paper studies the effect of chain-of-thought prompting, a recent approach that studies the steps followed by the model before it responds, on fairness. More specifically, we ask the following question: textitDo biased models have biased thoughts? To answer our question, we conduct experiments on 5 popular large language models using fairness metrics to quantify 11 different biases in the model's thoughts and output. Our results show that the bias in the thinking steps is not highly correlated with the output bias (less than 0.6 correlation with a p-value smaller than 0.001 in most cases). In other words, unlike human beings, the tested models with biased decisions do not always possess biased thoughts.