# The Bias is in the Details: An Assessment of Cognitive Bias in LLMs

arXiv: https://arxiv.org/abs/2509.22856 · HTML: https://mlsanity.com/paper/2509.22856/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 65 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family](https://mlsanity.com/paper/2303.07992/) (2023)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval](https://mlsanity.com/paper/2407.20371/) (2024)
- [A Comprehensive Evaluation of Cognitive Biases in LLMs](https://mlsanity.com/paper/2410.15413/) (2024)
- [Large Language Models Show Human-like Social Desirability Biases in Survey Responses](https://mlsanity.com/paper/2405.06058/) (2024)
- [Bias Unveiled: Investigating Social Bias in LLM-Generated Code](https://mlsanity.com/paper/2411.10351/) (2024)
- [Learning to Summarize from LLM-generated Feedback](https://mlsanity.com/paper/2410.13116/) (2024)
- [BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs](https://mlsanity.com/paper/2407.10241/) (2024)
- [Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesis](https://mlsanity.com/paper/2407.02030/) (2024)
- [AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment](https://mlsanity.com/paper/2409.16022/) (2024)
- [Cognitive Biases in Large Language Models: A Survey and Mitigation Experiments](https://mlsanity.com/paper/2412.00323/) (2024)
- [The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?](https://mlsanity.com/paper/2410.04699/) (2024)
- [Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs](https://mlsanity.com/paper/2404.01461/) (2024)
- [LLM-SAP: Large Language Models Situational Awareness Based Planning](https://mlsanity.com/paper/2312.16127/) (2023)
- [Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rights](https://mlsanity.com/paper/2502.19463/) (2025)

## Built on by (most cited first, 10 of 10)

- [Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions](https://mlsanity.com/paper/2605.00226/) (2026)
- [LLM-Based Agentic Negotiation for 6G: Addressing Uncertainty Neglect and Tail-Event Risk](https://mlsanity.com/paper/2511.19175/) (2025)
- [Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physics](https://mlsanity.com/paper/2605.10528/) (2026)
- [When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Making](https://mlsanity.com/paper/2603.18530/) (2026)
- [Bias by Necessity: Impossibility Theorems for Sequential Processing with Convergent AI and Human Validation](https://mlsanity.com/paper/2605.08716/) (2026)
- [Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer](https://mlsanity.com/paper/2609.28372/) (2026)
- [Exploring LLMs for South Asian Music Understanding and Generation](https://mlsanity.com/paper/2606.05522/) (2026)
- [Mitigating Anchoring Bias in LLM-Based Agents for Energy-Efficient 6G Autonomous Networks](https://mlsanity.com/paper/2606.18272/) (2026)
- [Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection](https://mlsanity.com/paper/2606.30587/) (2026)
- [The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Models](https://mlsanity.com/paper/2610.10049/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes, it becomes crucial to examine the extent to which they exhibit cognitive biases. Extensively studied in the field of psychology, cognitive biases appear as systematic distortions commonly observed in human judgments. This paper presents a large-scale evaluation of eight well-established cognitive biases across 45 LLMs, analyzing over 2.8 million LLM responses generated through controlled prompt variations. To achieve this, we introduce a novel evaluation framework based on multiple-choice tasks, hand-curate a dataset of 220 decision scenarios targeting fundamental cognitive biases in collaboration with psychologists, and propose a scalable approach for generating diverse prompts from human-authored scenario templates. Our analysis shows that LLMs exhibit bias-consistent behavior in 17.8-57.3% of instances across a range of judgment and decision-making contexts targeting anchoring, availability, confirmation, framing, interpretation, overattribution, prospect theory, and representativeness biases. We find that both model size and prompt specificity play a significant role on bias susceptibility as follows: larger size (>32B parameters) can reduce bias in 39.5% of cases, while higher prompt detail reduces most biases by up to 14.9%, except in one case (Overattribution), which is exacerbated by up to 8.8%.