mlsanityData to 7 Oct 2026

CBEval: A framework for evaluating and interpreting cognitive biases in LLMs

arXiv 2412.03605 ↗First posted 4 Dec 2024cs.CL cs.AI cs.HCcited by 12

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Rapid advancements in Large Language models (LLMs) has significantly enhanced their reasoning capabilities. Despite improved performance on benchmarks, LLMs exhibit notable gaps in their cognitive processes. Additionally, as reflections of human-generated data, these models have the potential to inherit cognitive biases, raising concerns about their reasoning and decision making capabilities. In this paper we present a framework to interpret, understand and provide insights into a host of cognitive biases in LLMs. Conducting our research on frontier language models we're able to elucidate reasoning limitations and biases, and provide reasoning behind these biases by constructing influence graphs that identify phrases and words most responsible for biases manifested in LLMs. We further investigate biases such as round number bias and cognitive bias barrier revealed when noting framing effect in language models.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 28 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

20 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners16 Jun

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  8. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  9. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  10. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  11. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  12. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
7 more
  1. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  2. 2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
  3. 2023Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoptioncited by 21
  4. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  5. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  6. 2024Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMscited by 12
  7. 2023Cognitive network science reveals bias in GPT-3, ChatGPT, and GPT-4 mirroring math anxiety in high-school studentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 3.

12 papers.

  1. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  2. 2025TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Modelscited by 31
  3. 2025Comparing the Framing Effect in Humans and LLMs on Naturally Occurring Textscited by 6
  4. 2025Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigationscited by 6
  5. 2025Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMscited by 5
  6. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 4
  7. 2025Towards Understanding the Cognitive Habits of Large Reasoning Modelscited by 4
  8. 2025Do Language Models Agree with Human Perceptions of Suspense in Stories?cited by 1
  9. 2026The Paradox of Robustness: Decoupling Rule-Based Logic from Affective Noise in High-Stakes Decision-Makingcited by 1
  10. 2026AMEL: Accumulated Message Effects on LLM Judgmentscited by 1
  11. 2025The Idola Tribus of AI: Large Language Models tend to perceive order where none existscited by 0
  12. 2026Survive at All Costs: Exploring LLM's Risky Behaviors under Survival Pressurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Biases in the Blind Spot: Detecting What LLMs Fail to Mention
  2. 2026BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs
  3. 2024Anchoring Bias in Large Language Models: An Experimental Study
  4. 2024(Ir)rationality and Cognitive Biases in Large Language Models
  5. 2025Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges
  6. 2024A Multi-LLM Debiasing Framework
  7. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  8. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning