# Cognitive Biases in Large Language Models: A Survey and Mitigation Experiments

arXiv: https://arxiv.org/abs/2412.00323 · HTML: https://mlsanity.com/paper/2412.00323/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 30 Nov 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5](https://mlsanity.com/paper/2305.04400/) (2023)
- [Addressing cognitive bias in medical language models](https://mlsanity.com/paper/2402.08113/) (2024)
- [Primacy Effect of ChatGPT](https://mlsanity.com/paper/2310.13206/) (2023)
- [(Ir)rationality and Cognitive Biases in Large Language Models](https://mlsanity.com/paper/2402.09193/) (2024)
- [Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption](https://mlsanity.com/paper/2304.01358/) (2023)
- [Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?](https://mlsanity.com/paper/2401.18070/) (2024)
- [Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns](https://mlsanity.com/paper/2305.04812/) (2023)
- [Large Language Models are Not Stable Recommender Systems](https://mlsanity.com/paper/2312.15746/) (2023)
- [Redefining "Hallucination" in LLMs: Towards a psychology-informed framework for mitigating misinformation](https://mlsanity.com/paper/2402.01769/) (2024)
- [HANS, are you clever? Clever Hans Effect Analysis of Neural Systems](https://mlsanity.com/paper/2309.12481/) (2023)

## Built on by (most cited first, 20 of 20)

- [Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs](https://mlsanity.com/paper/2505.11556/) (2025)
- [Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations](https://mlsanity.com/paper/2502.01349/) (2025)
- [The Bias is in the Details: An Assessment of Cognitive Bias in LLMs](https://mlsanity.com/paper/2509.22856/) (2025)
- [CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments](https://mlsanity.com/paper/2509.13588/) (2025)
- [Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Models](https://mlsanity.com/paper/2601.23255/) (2026)
- [How Does Cognitive Bias Affect Large Language Models? A Case Study on the Anchoring Effect in Price Negotiation Simulations](https://mlsanity.com/paper/2508.21137/) (2025)
- [Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM-based Manipulation Checks](https://mlsanity.com/paper/2503.00024/) (2025)
- [Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning](https://mlsanity.com/paper/2512.02914/) (2025)
- [Large Language Newsvendor: Decision Biases and Cognitive Mechanisms](https://mlsanity.com/paper/2512.12552/) (2025)
- [Anchors in the Machine: Behavioral and Attributional Evidence of Anchoring Bias in LLMs](https://mlsanity.com/paper/2511.05766/) (2025)
- [Talent or Luck? Evaluating Attribution Bias in Large Language Models](https://mlsanity.com/paper/2505.22910/) (2025)
- [Truth or Tribe: How In-group Favoritism Prioritize Facts in Persona Agents](https://mlsanity.com/paper/2605.01329/) (2026)
- [AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs](https://mlsanity.com/paper/2608.14320/) (2026)
- [AMEL: Accumulated Message Effects on LLM Judgments](https://mlsanity.com/paper/2605.22714/) (2026)
- [Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence](https://mlsanity.com/paper/2608.25869/) (2026)
- [Gauging Overprecision in LLMs: An Empirical Study](https://mlsanity.com/paper/2504.12098/) (2025)
- [VeriMinder: Mitigating Analytical Vulnerabilities in NL2SQL](https://mlsanity.com/paper/2507.17896/) (2025)
- [Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Models](https://mlsanity.com/paper/2605.24755/) (2026)
- [PraMem: Practice-derived Experiential Memory for Long-horizon Behavior Prediction](https://mlsanity.com/paper/2607.02881/) (2026)
- [Measuring Human-Like Bias in LLMs? A Critique of Human-Derived Bias Constructs in LLM Evaluation](https://mlsanity.com/paper/2610.00070/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are trained on large corpora written by humans and demonstrate high performance on various tasks. However, as humans are susceptible to cognitive biases, which can result in irrational judgments, LLMs can also be influenced by these biases, leading to irrational decision-making. For example, changing the order of options in multiple-choice questions affects the performance of LLMs due to order bias. In our research, we first conducted an extensive survey of existing studies examining LLMs' cognitive biases and their mitigation. The mitigation techniques in LLMs have the disadvantage that they are limited in the type of biases they can apply or require lengthy inputs or outputs. We then examined the effectiveness of two mitigation methods for humans, SoPro and AwaRe, when applied to LLMs, inspired by studies in crowdsourcing. To test the effectiveness of these methods, we conducted experiments on GPT-3.5 and GPT-4 to evaluate the influence of six biases on the outputs before and after applying these methods. The results demonstrate that while SoPro has little effect, AwaRe enables LLMs to mitigate the effect of these biases and make more rational responses.