mlsanityData to 7 Oct 2026

Thinking Fast and Slow in Large Language Models

arXiv 2212.05206 ↗First posted 10 Dec 2022cs.CL cs.AI cs.LGcited by 136

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. Therefore, it is of great importance to evaluate their emerging abilities. In this study, we show that LLMs like GPT-3 exhibit behavior that strikingly resembles human-like intuition - and the cognitive errors that come with it. However, LLMs with higher cognitive capabilities, in particular ChatGPT and GPT-4, learned to avoid succumbing to these errors and perform in a hyperrational manner. For our experiments, we probe LLMs with the Cognitive Reflection Test (CRT) as well as semantic illusions that were originally designed to investigate intuitive decision-making in humans. Our study demonstrates that investigating LLMs with methods from psychology has the potential to reveal otherwise unknown emergent traits.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 60 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

12 of 60 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2022Using cognitive psychology to understand GPT-3cited by 389
  10. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  11. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  12. 2020An Analysis of Dataset Overlap on Winograd-Style Taskscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 47; 2025: 44; 2026: 31.

136 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  3. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  4. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  6. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  7. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  8. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  9. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  10. 2024CogBench: a large language model walks into a psychology labcited by 60
  11. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  12. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
38 more
  1. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
  2. 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 49
  3. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
  4. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 46
  5. 2024Y Social: an LLM-powered Social Media Digital Twincited by 43
  6. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  7. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  8. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  9. 2023Visual cognition in multimodal large language modelscited by 38
  10. 2024Prompting Techniques for Reducing Social Bias in LLMs through System 1 and System 2 Cognitive Processescited by 38
  11. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  12. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  13. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  14. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrailscited by 29
  15. 2023LLMind: Orchestrating AI and IoT with LLM for Complex Task Executioncited by 24
  16. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  17. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24
  18. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  19. 2023The Machine Psychology of Cooperation: Can GPT models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games?cited by 22
  20. 2024Toward LLM-Agent-Based Modeling of Transportation Systems: A Conceptual Frameworkcited by 19
  21. 2024Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Reviewcited by 17
  22. 2023Machine Psychologycited by 16
  23. 2024Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Taskscited by 15
  24. 2024Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Modelscited by 14
  25. 2024Designing AI-Agents with Personalities: A Psychometric Approachcited by 14
  26. 2025Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinkingcited by 14
  27. 2025When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMscited by 14
  28. 2024Do LLMs have Consistent Values?cited by 13
  29. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  30. 2025The Greatest Good Benchmark: Measuring LLMs' Alignment with Utilitarian Moral Dilemmascited by 12
  31. 2024Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choicecited by 10
  32. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  33. 2024Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillationcited by 9
  34. 2024Preemptive Detection and Correction of Misaligned Actions in LLM Agentscited by 9
  35. 2025Spontaneous Giving and Calculated Greed in Language Modelscited by 9
  36. 2024Effective Generative AI: The Human-Algorithm Centaurcited by 8
  37. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  38. 2025How Personality Traits Shape LLM Risk-Taking Behaviourcited by 8

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models -- and Disappeared in GPT-4
  2. 2026Understanding Large Language Models
  3. 2024Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models
  4. 2026Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?
  5. 2025Large Language Models Think Too Fast To Explore Effectively
  6. 2023Large Language Models are biased to overestimate profoundness
  7. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
  8. 2025Do Large Language Models Know What They Are Capable Of?