mlsanityData to 7 Oct 2026

Using cognitive psychology to understand GPT-3

arXiv 2206.14576 ↗First posted 21 Jun 2022cs.CL cs.AI cs.LGcited by 389

Not on the map. Why?

Cited by 44 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

Key facts. Cited by 389 papers in the map (data through 7 Oct 2026). Builds on 24 of its 75 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

24 of 75 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  9. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  10. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  11. 2022Can language models learn from explanations in context?cited by 248
  12. 2019Causality for Machine Learningcited by 242
12 more
  1. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  2. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  3. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  4. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  5. 2019Causal Reasoning from Meta-reinforcement Learningcited by 53
  6. 2019Environmental drivers of systematicity and generalization in a situated agentcited by 50
  7. 2017Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Studycited by 46
  8. 2020The Chess Transformer: Mastering Play using Generative Language Modelscited by 36
  9. 2021Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2cited by 15
  10. 2022Towards Understanding How Machines Can Learn Causal Overhypothesescited by 13
  11. 2020A neural network walks into a lab: towards using deep nets as models for human behaviorcited by 10
  12. 2021Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humanscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 84; 2024: 114; 2025: 90; 2026: 88.

389 papers.

Sort:

By area:

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2023Whose Opinions Do Language Models Reflect?cited by 759
  5. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  6. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  7. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  8. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  9. 2023Playing repeated games with Large Language Modelscited by 213
  10. 2022Language models show human-like content effects on reasoning taskscited by 171
  11. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  12. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
38 more
  1. 2023Simulating Opinion Dynamics with Networks of LLM-based Agentscited by 153
  2. 2025Do generative video models understand physical principles?cited by 151
  3. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  4. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  5. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  6. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 128
  7. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  8. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  9. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  10. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  11. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  12. 2024Can large language models explore in-context?cited by 75
  13. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  14. 2023Turning large language models into cognitive modelscited by 66
  15. 2024CogBench: a large language model walks into a psychology labcited by 60
  16. 2023Meta-in-context learning in large language modelscited by 59
  17. 2024Can Language Models Learn to Skip Steps?cited by 59
  18. 2023On the Creativity of Large Language Modelscited by 57
  19. 2023Do LLM Agents Exhibit Social Behavior?cited by 53
  20. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
  21. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 49
  22. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  23. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  24. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 48
  25. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
  26. 2023Large language models predict human sensory judgments across six modalitiescited by 47
  27. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 46
  28. 2024People cannot distinguish GPT-4 from a human in a Turing testcited by 46
  29. 2023Diminished Diversity-of-Thought in a Standard Large Language Modelcited by 43
  30. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  31. 2023Visual cognition in multimodal large language modelscited by 38
  32. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  33. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 35
  34. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  35. 2023Mind meets machine: Unravelling GPT-4's cognitive psychologycited by 31
  36. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 31
  37. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  38. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 30

50 most cited of 389 All 389 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
  2. 2026Reasoning Promotes Robustness in Theory of Mind Tasks
  3. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  4. 2026Reasoning aligns language models to human cognition
  5. 2024Mental Modeling of Reinforcement Learning Agents by Language Models
  6. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures
  7. 2025Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models
  8. 2026Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning