mlsanityData to 7 Oct 2026

Capturing Failures of Large Language Models via Human Cognitive Biases

arXiv 2202.12299 ↗First posted 24 Feb 2022cs.CL cs.AI cs.LGcited by 97

Not on the map. Why?

Abstract (arXiv)

Large language models generate complex, open-ended outputs: instead of outputting a class label they write summaries, generate dialogue, or produce working code. In order to asses the reliability of these open-ended generation systems, we aim to identify qualitative categories of erroneous behavior, beyond identifying individual errors. To hypothesize and test for such qualitative errors, we draw inspiration from human cognitive biases -- systematic patterns of deviation from rational judgement. Specifically, we use cognitive biases as motivation to (i) generate hypotheses for problems that models may have, and (ii) develop experiments that elicit these problems. Using code generation as a case study, we find that OpenAI's Codex errs predictably based on how the input prompt is framed, adjusts outputs towards anchors, and is biased towards outputs that mimic frequent training examples. We then use our framework to elicit high-impact errors such as incorrectly deleting files. Our results indicate that experimental methodology from cognitive science can help characterize how machine learning systems behave.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

34 of 69 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
22 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  12. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  13. 2019ELI5: Long Form Question Answeringcited by 567
  14. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  15. 2019Measurement and Fairnesscited by 299
  16. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 215
  17. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  18. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  19. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 75
  20. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  21. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 52
  22. 2021Solving Probability and Statistics Problems by Program Synthesiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 27; 2024: 25; 2025: 21; 2026: 21.

97 papers.

Sort:

By area:

  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  3. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  5. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  6. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  7. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  8. 2023Conformal Language Modelingcited by 163
  9. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  10. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  11. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  12. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
38 more
  1. 2024Cognitive Bias in Decision-Making with LLMscited by 104
  2. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  3. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  4. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  5. 2023LoRA ensembles for large language model fine-tuningcited by 65
  6. 2024Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrievalcited by 49
  7. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
  8. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  9. 2023Supporting Human-AI Collaboration in Auditing LLMs with LLMscited by 46
  10. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  11. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  12. 2024AI for social science and social science of AI: A Surveycited by 29
  13. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  14. 2024Cognitive Biases in Large Language Models: A Survey and Mitigation Experimentscited by 20
  15. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  16. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  17. 2025Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Modelscited by 19
  18. 2023Machine Psychologycited by 16
  19. 2024Large Language Model Benchmarks in Medical Taskscited by 16
  20. 2023Unraveling Downstream Gender Bias from Large Language Models: A Study on AI Educational Writing Assistancecited by 15
  21. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  22. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  23. 2023Cognitive Effects in Large Language Modelscited by 14
  24. 2023Quantifying and Attributing the Hallucination of Large Language Models via Association Analysiscited by 14
  25. 2023Improving Interpersonal Communication by Simulating Audiences with Language Modelscited by 14
  26. 2024Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Modelscited by 14
  27. 2023Assessing the Reliability of Large Language Model Knowledgecited by 13
  28. 2023Towards the Scalable Evaluation of Cooperativeness in Language Modelscited by 12
  29. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  30. 2024CBEval: A framework for evaluating and interpreting cognitive biases in LLMscited by 12
  31. 2025Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendationscited by 10
  32. 2023The Confidence-Competence Gap in Large Language Models: A Cognitive Studycited by 9
  33. 2023Using Artificial Populations to Study Psychological Phenomena in Neural Modelscited by 8
  34. 2024Large Language Model Recall Uncertainty is Modulated by the Fan Effectcited by 7
  35. 2023Recursive Visual Programmingcited by 6
  36. 2024Cognitive Biases in Large Language Models for News Recommendationcited by 6
  37. 2025Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigationscited by 6
  38. 2025Quantifying Cognitive Bias Induction in LLM-Generated Contentcited by 6

50 most cited of 97 All 97 (JSON)