# A Comprehensive Evaluation of Cognitive Biases in LLMs

arXiv: https://arxiv.org/abs/2410.15413 · HTML: https://mlsanity.com/paper/2410.15413/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 227 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [The Political Preferences of LLMs](https://mlsanity.com/paper/2402.01789/) (2024)
- [The Turking Test: Can Language Models Understand Instructions?](https://mlsanity.com/paper/2010.11982/) (2020)
- [Can Large Language Models Understand Real-World Complex Instructions?](https://mlsanity.com/paper/2309.09150/) (2023)

## Built on by (most cited first, 30 of 34)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Whose Facts Win? LLM Source Preferences under Knowledge Conflicts](https://mlsanity.com/paper/2601.03746/) (2026)
- [Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations](https://mlsanity.com/paper/2502.01349/) (2025)
- [The Bias is in the Details: An Assessment of Cognitive Bias in LLMs](https://mlsanity.com/paper/2509.22856/) (2025)
- [Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models](https://mlsanity.com/paper/2604.02485/) (2026)
- [Comparing the Framing Effect in Humans and LLMs on Naturally Occurring Texts](https://mlsanity.com/paper/2502.17091/) (2025)
- [Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models](https://mlsanity.com/paper/2506.14092/) (2025)
- [Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs](https://mlsanity.com/paper/2507.07186/) (2025)
- [Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts](https://mlsanity.com/paper/2508.06361/) (2025)
- [Large Language Newsvendor: Decision Biases and Cognitive Mechanisms](https://mlsanity.com/paper/2512.12552/) (2025)
- [Mechanisms of Prompt-Induced Hallucination in Vision-Language Models](https://mlsanity.com/paper/2601.05201/) (2026)
- [Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Making](https://mlsanity.com/paper/2504.04141/) (2025)
- [Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review](https://mlsanity.com/paper/2603.18740/) (2026)
- [AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs](https://mlsanity.com/paper/2608.14320/) (2026)
- [AI Through the Human Lens: Investigating Cognitive Theories in Machine Psychology](https://mlsanity.com/paper/2506.18156/) (2025)
- [Do Language Models Agree with Human Perceptions of Suspense in Stories?](https://mlsanity.com/paper/2508.15794/) (2025)
- [The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning](https://mlsanity.com/paper/2603.29025/) (2026)
- [One Size Fits None: Heuristic Collapse in LLM Investment Advice](https://mlsanity.com/paper/2604.23837/) (2026)
- [Evidence of conceptual mastery in the application of rules by Large Language Models](https://mlsanity.com/paper/2503.00992/) (2025)
- [The Prompting Brain: Neurocognitive Markers of Expertise in Guiding Large Language Models](https://mlsanity.com/paper/2508.14869/) (2025)
- [Shachi: A Modular, Controllable Framework for LLM-Based Agent-Based Modeling of Emergent Collective Behavior](https://mlsanity.com/paper/2509.21862/) (2025)
- [Predicting Biased Human Decision-Making with Large Language Models in Conversational Settings](https://mlsanity.com/paper/2601.11049/) (2026)
- [Emulating Aggregate Human Choice Behavior and Biases with GPT Conversational Agents](https://mlsanity.com/paper/2602.05597/) (2026)
- [CogBias: Measuring and Mitigating Cognitive Bias in Large Language Models](https://mlsanity.com/paper/2604.01366/) (2026)
- [Mitigating Cognitive Bias in RLHF by Altering Rationality](https://mlsanity.com/paper/2605.06895/) (2026)
- [Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection](https://mlsanity.com/paper/2606.30587/) (2026)
- [Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenders](https://mlsanity.com/paper/2607.10235/) (2026)
- [OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment](https://mlsanity.com/paper/2607.26981/) (2026)
- [RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation](https://mlsanity.com/paper/2608.01810/) (2026)
- [Conditional Cognitive Biases in LLMs: How Biased User Turns Modulate In-Context Reasoning](https://mlsanity.com/paper/2608.05166/) (2026)

## Abstract (arXiv)

We present a large-scale evaluation of 30 cognitive biases in 20 state-of-the-art large language models (LLMs) under various decision-making scenarios. Our contributions include a novel general-purpose test framework for reliable and large-scale generation of tests for LLMs, a benchmark dataset with 30,000 tests for detecting cognitive biases in LLMs, and a comprehensive assessment of the biases found in the 20 evaluated LLMs. Our work confirms and broadens previous findings suggesting the presence of cognitive biases in LLMs by reporting evidence of all 30 tested biases in at least some of the 20 LLMs. We publish our framework code to encourage future research on biases in LLMs: https://github.com/simonmalberg/cognitive-biases-in-llms