# Biases in the Blind Spot: Detecting What LLMs Fail to Mention

arXiv: https://arxiv.org/abs/2602.10117 · HTML: https://mlsanity.com/paper/2602.10117/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 49 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [Faithfulness Tests for Natural Language Explanations](https://mlsanity.com/paper/2305.18029/) (2023)
- [Are DeepSeek R1 And Other Reasoning Models More Faithful?](https://mlsanity.com/paper/2501.08156/) (2025)
- [Measuring Implicit Bias in Explicitly Unbiased Large Language Models](https://mlsanity.com/paper/2402.04105/) (2024)
- [The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models](https://mlsanity.com/paper/2404.03189/) (2024)
- [Robustly Improving LLM Fairness in Realistic Settings via Interpretability](https://mlsanity.com/paper/2506.10922/) (2025)
- [LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations](https://mlsanity.com/paper/2509.09396/) (2025)
- [A Causal Lens for Evaluating Faithfulness Metrics](https://mlsanity.com/paper/2502.18848/) (2025)
- [Detecting Prefix Bias in LLM-based Reward Models](https://mlsanity.com/paper/2505.13487/) (2025)
- [Verbosity Tradeoffs and the Impact of Scale on the Faithfulness of LLM Self-Explanations](https://mlsanity.com/paper/2503.13445/) (2025)
- [Muslim-Violence Bias Persists in Debiased GPT Models](https://mlsanity.com/paper/2310.18368/) (2023)
- [With a Grain of SALT: Are LLMs Fair Across Social Dimensions?](https://mlsanity.com/paper/2410.12499/) (2024)

## Built on by (most cited first, 8 of 8)

- [Counterfactual Simulation Training for Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2602.20710/) (2026)
- [Asymmetric Goal Drift in Coding Agents Under Value Conflict](https://mlsanity.com/paper/2603.03456/) (2026)
- [Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth](https://mlsanity.com/paper/2605.25052/) (2026)
- [Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settings](https://mlsanity.com/paper/2608.04735/) (2026)
- [Chain-of-Thought Faithfulness of Reasoning Models Varies with Where and How Preference Cues Are Delivered](https://mlsanity.com/paper/2608.29464/) (2026)
- [Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature](https://mlsanity.com/paper/2606.05206/) (2026)
- [Training Large Language Models for Self-Explanation Faithfulness](https://mlsanity.com/paper/2607.21090/) (2026)
- [On the Generalization of Steering Vectors for Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2607.29062/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) often provide chain-of-thought (CoT) reasoning traces that appear plausible, but may hide internal biases. We call these unverbalized biases. Monitoring models via their stated reasoning is therefore unreliable, and existing bias evaluations typically require predefined categories and hand-crafted datasets. In this work, we introduce a fully automated, black-box pipeline for detecting task-specific unverbalized biases. Given a task dataset, the pipeline uses LLM autoraters to generate candidate bias concepts. It then tests each concept on progressively larger input samples by generating positive and negative variations, and applies statistical techniques for multiple testing and early stopping. A concept is flagged as an unverbalized bias if it yields statistically significant performance differences while not being cited as justification in the model's CoTs. We evaluate our pipeline across seven LLMs on three decision tasks (hiring, loan approval, and university admissions). Our technique automatically discovers previously unknown biases in these models (e.g., Spanish fluency, English proficiency, writing formality). In the same run, the pipeline also validates biases that were manually identified by prior work (gender, race, religion, ethnicity). More broadly, our proposed approach provides a practical, scalable path to automatic, more efficient, and broader task-specific unverbalized bias discovery.