# CBEval: A framework for evaluating and interpreting cognitive biases in LLMs

arXiv: https://arxiv.org/abs/2412.03605 · HTML: https://mlsanity.com/paper/2412.03605/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners](https://mlsanity.com/paper/2406.11050/) (2024)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization](https://mlsanity.com/paper/2405.17067/) (2024)
- [Creativity Has Left the Chat: The Price of Debiasing Language Models](https://mlsanity.com/paper/2406.05587/) (2024)
- [Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption](https://mlsanity.com/paper/2304.01358/) (2023)
- [Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?](https://mlsanity.com/paper/2401.18070/) (2024)
- [Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs](https://mlsanity.com/paper/2404.01461/) (2024)
- [Explaining Large Language Models Decisions Using Shapley Values](https://mlsanity.com/paper/2404.01332/) (2024)
- [Cognitive network science reveals bias in GPT-3, ChatGPT, and GPT-4 mirroring math anxiety in high-school students](https://mlsanity.com/paper/2305.18320/) (2023)

## Built on by (most cited first, 12 of 12)

- [Improving LLM-as-a-Judge Inference with the Judgment Distribution](https://mlsanity.com/paper/2503.03064/) (2025)
- [TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models](https://mlsanity.com/paper/2503.11656/) (2025)
- [Comparing the Framing Effect in Humans and LLMs on Naturally Occurring Texts](https://mlsanity.com/paper/2502.17091/) (2025)
- [Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigations](https://mlsanity.com/paper/2505.15392/) (2025)
- [Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models](https://mlsanity.com/paper/2506.14092/) (2025)
- [Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs](https://mlsanity.com/paper/2507.07186/) (2025)
- [Towards Understanding the Cognitive Habits of Large Reasoning Models](https://mlsanity.com/paper/2506.21571/) (2025)
- [Do Language Models Agree with Human Perceptions of Suspense in Stories?](https://mlsanity.com/paper/2508.15794/) (2025)
- [The Paradox of Robustness: Decoupling Rule-Based Logic from Affective Noise in High-Stakes Decision-Making](https://mlsanity.com/paper/2601.21439/) (2026)
- [AMEL: Accumulated Message Effects on LLM Judgments](https://mlsanity.com/paper/2605.22714/) (2026)
- [The Idola Tribus of AI: Large Language Models tend to perceive order where none exists](https://mlsanity.com/paper/2510.09709/) (2025)
- [Survive at All Costs: Exploring LLM's Risky Behaviors under Survival Pressure](https://mlsanity.com/paper/2603.05028/) (2026)

## Abstract (arXiv)

Rapid advancements in Large Language models (LLMs) has significantly enhanced their reasoning capabilities. Despite improved performance on benchmarks, LLMs exhibit notable gaps in their cognitive processes. Additionally, as reflections of human-generated data, these models have the potential to inherit cognitive biases, raising concerns about their reasoning and decision making capabilities. In this paper we present a framework to interpret, understand and provide insights into a host of cognitive biases in LLMs. Conducting our research on frontier language models we're able to elucidate reasoning limitations and biases, and provide reasoning behind these biases by constructing influence graphs that identify phrases and words most responsible for biases manifested in LLMs. We further investigate biases such as round number bias and cognitive bias barrier revealed when noting framing effect in language models.