mlsanityData to 7 Oct 2026

CogBench: a large language model walks into a psychology lab

arXiv 2402.18225 ↗First posted 28 Feb 2024cs.CL cs.AI cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 35 LLMs, yielding a rich and diverse dataset. We analyze this data using statistical multilevel modeling techniques, accounting for the nested dependencies among fine-tuned versions of specific LLMs. Our study highlights the crucial role of model size and reinforcement learning from human feedback (RLHF) in improving performance and aligning with human behavior. Interestingly, we find that open-source models are less risk-prone than proprietary models and that fine-tuning on code does not necessarily enhance LLMs' behavior. Finally, we explore the effects of prompt-engineering techniques. We discover that chain-of-thought prompting improves probabilistic reasoning, while take-a-step-back prompting fosters model-based behaviors.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 34 of its 66 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

34 of 66 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
22 more
  1. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  7. 2022Using cognitive psychology to understand GPT-3cited by 389
  8. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  9. 2022Can language models learn from explanations in context?cited by 248
  10. 2023Playing repeated games with Large Language Modelscited by 213
  11. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  12. 2022Language models show human-like content effects on reasoning taskscited by 171
  13. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  14. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  15. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  16. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  17. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
  18. 2023Meta-in-context learning in large language modelscited by 59
  19. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  20. 2023Prompting Frameworks for Large Language Models: A Surveycited by 29
  21. 2023Studying and improving reasoning in humans and machinescited by 14
  22. 2023How should the advent of large language models affect the practice of science?cited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 26; 2026: 15.

60 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  3. 2024Can large language models explore in-context?cited by 75
  4. 2024Can Language Models Learn to Skip Steps?cited by 59
  5. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  6. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  7. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
  8. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Modelscited by 35
  9. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  10. 2024TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Studentscited by 25
  11. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24
  12. 2024When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settingscited by 17
38 more
  1. 2024Large Language Model Benchmarks in Medical Taskscited by 16
  2. 2024Can LLMs make trade-offs involving stipulated pain and pleasure states?cited by 16
  3. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  4. 2024M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmarkcited by 14
  5. 2025Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systemscited by 13
  6. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  7. 2025The Greatest Good Benchmark: Measuring LLMs' Alignment with Utilitarian Moral Dilemmascited by 12
  8. 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
  9. 2025Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agentscited by 11
  10. 2024Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choicecited by 10
  11. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  12. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 10
  13. 2025Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answerscited by 10
  14. 2025Efficient Ensemble for Fine-tuning Language Models on Multiple Datasetscited by 10
  15. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  16. 2025Levels of Analysis for Large Language Modelscited by 9
  17. 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
  18. 2025Should You Use Your Large Language Model to Explore or Exploit?cited by 8
  19. 2024Large Language Model Recall Uncertainty is Modulated by the Fan Effectcited by 7
  20. 2025Are Large Language Models Sensitive to the Motives Behind Communication?cited by 7
  21. 2025Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMscited by 6
  22. 2024Large Language Models are Biased Reinforcement Learnerscited by 5
  23. 2025AI Agent Behavioral Sciencecited by 5
  24. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  25. 2026A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilitiescited by 4
  26. 2025LLM Agents Display Human Biases but Exhibit Distinct Learning Patternscited by 3
  27. 2025H2HTalk: Evaluating Large Language Models as Emotional Companioncited by 3
  28. 2025How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMscited by 3
  29. 2025How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn Conversationscited by 3
  30. 2025The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theoriescited by 2
  31. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  32. 2025Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulationcited by 2
  33. 2026AI as Entertainmentcited by 2
  34. 2026Discovering Differences in Strategic Behavior Between Humans and LLMscited by 2
  35. 2026NeuReasoner: Theory-grounded Mapping of Reasoning Elicitation Boundariescited by 2
  36. 2024Meaningless is better: hashing bias-inducing words in LLM prompts improves performance in logical reasoning and statistical learningcited by 1
  37. 2025Unraveling the cognitive patterns of Large Language Models through module communitiescited by 0
  38. 2026Language Model Goal Selection Differs from Humans' in a Self-Directed Learning Taskcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Turning large language models into cognitive models
  2. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions
  3. 2023Machine Psychology
  4. 2026Understanding Large Language Models
  5. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  6. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
  7. 2024LLM2: Let Large Language Models Harness System 2 Reasoning
  8. 2025Reasoning Capabilities of Large Language Models on Dynamic Tasks