mlsanityData to 14 Oct 2026

The Bias is in the Details: An Assessment of Cognitive Bias in LLMs

arXiv 2509.22856 ↗First posted 26 Sep 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes, it becomes crucial to examine the extent to which they exhibit cognitive biases. Extensively studied in the field of psychology, cognitive biases appear as systematic distortions commonly observed in human judgments. This paper presents a large-scale evaluation of eight well-established cognitive biases across 45 LLMs, analyzing over 2.8 million LLM responses generated through controlled prompt variations. To achieve this, we introduce a novel evaluation framework based on multiple-choice tasks, hand-curate a dataset of 220 decision scenarios targeting fundamental cognitive biases in collaboration with psychologists, and propose a scalable approach for generating diverse prompts from human-authored scenario templates. Our analysis shows that LLMs exhibit bias-consistent behavior in 17.8-57.3% of instances across a range of judgment and decision-making contexts targeting anchoring, availability, confirmation, framing, interpretation, overattribution, prospect theory, and representativeness biases. We find that both model size and prompt specificity play a significant role on bias susceptibility as follows: larger size (>32B parameters) can reduce bias in 39.5% of cases, while higher prompt detail reduces most biases by up to 14.9%, except in one case (Overattribution), which is exacerbated by up to 8.8%.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 65 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

24 of 65 references.

References · 24

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  3. 2022Using cognitive psychology to understand GPT-3cited by 398
  4. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  5. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  6. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  7. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  8. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  9. 2024Cognitive Bias in Decision-Making with LLMscited by 106
  10. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  11. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 53
  12. 2024Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrievalcited by 51
12 more
  1. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 34
  2. 2024Large Language Models Show Human-like Social Desirability Biases in Survey Responsescited by 28
  3. 2024Bias Unveiled: Investigating Social Bias in LLM-Generated Codecited by 25
  4. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  5. 2024BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMscited by 21
  6. 2024Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesiscited by 20
  7. 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
  8. 2024Cognitive Biases in Large Language Models: A Survey and Mitigation Experimentscited by 20
  9. 2024The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?cited by 14
  10. 2024Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMscited by 13
  11. 2023LLM-SAP: Large Language Models Situational Awareness Based Planningcited by 4
  12. 2025Hedging and Non-Affirmation: Quantifying LLM Alignment on Questions of Human Rightscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 3
  2. 2025LLM-Based Agentic Negotiation for 6G: Addressing Uncertainty Neglect and Tail-Event Riskcited by 2
  3. 2026Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physicscited by 2
  4. 2026When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Makingcited by 1
  5. 2026Bias by Necessity: Impossibility Theorems for Sequential Processing with Convergent AI and Human Validationcited by 1
  6. 2026Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumercited by 1
  7. 2026Exploring LLMs for South Asian Music Understanding and Generationcited by 0
  8. 2026Mitigating Anchoring Bias in LLM-Based Agents for Energy-Efficient 6G Autonomous Networkscited by 0
  9. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0
  10. 2026The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Modelscited by 0