mlsanityData to 7 Oct 2026

A Comprehensive Evaluation of Cognitive Biases in LLMs

arXiv 2410.15413 ↗First posted 20 Oct 2024cs.CL cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

We present a large-scale evaluation of 30 cognitive biases in 20 state-of-the-art large language models (LLMs) under various decision-making scenarios. Our contributions include a novel general-purpose test framework for reliable and large-scale generation of tests for LLMs, a benchmark dataset with 30,000 tests for detecting cognitive biases in LLMs, and a comprehensive assessment of the biases found in the 20 evaluated LLMs. Our work confirms and broadens previous findings suggesting the presence of cognitive biases in LLMs by reporting evidence of all 30 tested biases in at least some of the 20 LLMs. We publish our framework code to encourage future research on biases in LLMs: https://github.com/simonmalberg/cognitive-biases-in-llms

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 43 of its 227 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

43 of 227 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2023BloombergGPT: A Large Language Model for Financecited by 923
  12. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
31 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  6. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  7. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  8. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  9. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  10. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  11. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  12. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  13. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  14. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  15. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  16. 2024The Political Preferences of LLMscited by 83
  17. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  18. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  19. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  20. 2024Controllable Text Generation for Large Language Models: A Surveycited by 55
  21. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
  22. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  23. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 46
  24. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  25. 2024Addressing cognitive bias in medical language modelscited by 34
  26. 2023Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Modelscited by 32
  27. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiencycited by 31
  28. 2024Metaphor Understanding Challenge Dataset for LLMscited by 30
  29. 2023Making Large Language Models Better Data Creatorscited by 28
  30. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  31. 2024Exploring Conversational Agents as an Effective Tool for Measuring Cognitive Biases in Decision-Makingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 19.

32 papers.

Sort:

By area:

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2025Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendationscited by 10
  3. 2025The Bias is in the Details: An Assessment of Cognitive Bias in LLMscited by 10
  4. 2026Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Modelscited by 9
  5. 2026Whose Facts Win? LLM Source Preferences under Knowledge Conflictscited by 8
  6. 2025Comparing the Framing Effect in Humans and LLMs on Naturally Occurring Textscited by 6
  7. 2025Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMscited by 5
  8. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Promptscited by 5
  9. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  10. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 4
  11. 2025Large Language Newsvendor: Decision Biases and Cognitive Mechanismscited by 4
  12. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
20 more
  1. 2026Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Reviewcited by 3
  2. 2026AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMscited by 2
  3. 2025AI Through the Human Lens: Investigating Cognitive Theories in Machine Psychologycited by 1
  4. 2025Do Language Models Agree with Human Perceptions of Suspense in Stories?cited by 1
  5. 2026The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoningcited by 1
  6. 2026One Size Fits None: Heuristic Collapse in LLM Investment Advicecited by 1
  7. 2025Evidence of conceptual mastery in the application of rules by Large Language Modelscited by 0
  8. 2025The Prompting Brain: Neurocognitive Markers of Expertise in Guiding Large Language Modelscited by 0
  9. 2025Shachi: A Modular, Controllable Framework for LLM-Based Agent-Based Modeling of Emergent Collective Behaviorcited by 0
  10. 2026Predicting Biased Human Decision-Making with Large Language Models in Conversational Settingscited by 0
  11. 2026Emulating Aggregate Human Choice Behavior and Biases with GPT Conversational Agentscited by 0
  12. 2026Mitigating Cognitive Bias in RLHF by Altering Rationalitycited by 0
  13. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0
  14. 2026Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenderscited by 0
  15. 2026OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgmentcited by 0
  16. 2026RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluationcited by 0
  17. 2026Conditional Cognitive Biases in LLMs: How Biased User Turns Modulate In-Context Reasoningcited by 0
  18. 2026Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Makingcited by 0
  19. 2026Measuring Human-Like Bias in LLMs? A Critique of Human-Derived Bias Constructs in LLM Evaluationcited by 0
  20. 2026The Long Road to the Same Answer: Cognitive Bias Under Escalating Reasoning Budgets in Large Language Modelscited by 0