mlsanityData to 14 Oct 2026

On Benchmarking Human-Like Intelligence in Machines

arXiv 2502.20502 ↗First posted 27 Feb 2025cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition. However, we argue that many current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities in these models. We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically invalid tasks. We support our claims by conducting a human evaluation study on nine existing AI benchmarks, suggesting major limitations in task and label designs. To address these limitations, we propose five concrete recommendations for future AI evaluation efforts that will enable more rigorous and meaningful understanding of human-like cognitive capacities in AI models.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 53 of its 134 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

53 of 134 references.

References · 53

  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  3. 2022Using cognitive psychology to understand GPT-3cited by 398
  4. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  5. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  6. 2023Role-Play with Large Language Modelscited by 294
  7. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  8. 2019Human uncertainty makes classification more robustcited by 235
  9. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 222
  10. 2024A Roadmap to Pluralistic Alignmentcited by 184
  11. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  12. 2023Personality Traits in Large Language Modelscited by 169
38 more
  1. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  2. 2023Getting aligned on representational alignmentcited by 143
  3. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  4. 2022Thinking Fast and Slow in Large Language Modelscited by 137
  5. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  6. 2022Evaluating Human-Language Model Interactioncited by 92
  7. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  8. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  9. 2022Human alignment of neural network representationscited by 74
  10. 2024Building Machines that Learn and Think with Peoplecited by 73
  11. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
  12. 2023Evaluating Language Models for Mathematics through Interactionscited by 70
  13. 2024CogBench: a large language model walks into a psychology labcited by 63
  14. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 62
  15. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  16. 2023Rehearsal: Simulating Conflict to Teach Conflict Resolutioncited by 55
  17. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  18. 2024Large Language Models Assume People are More Rational than We Really arecited by 54
  19. 2024Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertisecited by 50
  20. 2023Prompting is not a substitute for probability measurements in large language modelscited by 44
  21. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  22. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  23. 2023Human Uncertainty in Concept-Based AI Systemscited by 39
  24. 2023Improving neural network representations using human similarity judgmentscited by 38
  25. 2020Understanding Human Intelligence through Human Limitationscited by 33
  26. 2020Gibbs Sampling with Peoplecited by 28
  27. 2024Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planningcited by 28
  28. 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
  29. 2019Towards a Quantum-Like Cognitive Architecture for Decision-Makingcited by 17
  30. 2023Inferring the Goals of Communicating Agents from Actions and Instructionscited by 17
  31. 2019The Animal-AI Environment: Training and Testing Animal-Like Artificial Cognitioncited by 16
  32. 2020'Less Than One'-Shot Learning: Learning N Classes From M<N Samplescited by 15
  33. 2023Improving Interpersonal Communication by Simulating Audiences with Language Modelscited by 14
  34. 2022On the Informativeness of Supervision Signalscited by 12
  35. 2024Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learningcited by 11
  36. 2022Human-in-the-Loop Mixupcited by 8
  37. 2023Can AI Serve as a Substitute for Human Subjects in Software Engineering Research?cited by 8
  38. 2024Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mindcited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 7.

25 papers.

Sort:

By area:

  1. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 52
  2. 2025BEARCUBS: A benchmark for computer-using web agentscited by 31
  3. 2025Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspectivecited by 14
  4. 2025Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systemscited by 14
  5. 2025Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Modelscited by 14
  6. 2025Simulating Society Requires Simulating Thoughtcited by 13
  7. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  8. 2026HumanLLM: Towards Personalized Understanding and Simulation of Human Naturecited by 8
  9. 2025Using LLMs to Advance the Cognitive Science of Collectivescited by 6
  10. 2025Language-Informed Synthesis of Rational Agent Models for Grounded Theory-of-Mind Reasoning On-The-Flycited by 6
  11. 2025On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Conceptscited by 5
  12. 2025A large-scale evaluation of commonsense knowledge in humans and large language modelscited by 4
13 more
  1. 2025HugAgent: A Human Simulation Benchmark for Individual-Level Reasoningcited by 2
  2. 2026HumanStudy-Bench: Towards AI Agent Design for Participant Simulationcited by 2
  3. 2025V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chatcited by 1
  4. 2025NiceWebRL: a Python library for human subject experiments with reinforcement learning environmentscited by 1
  5. 2025Uncovering the Computational Ingredients of Human-Like Representations in LLMscited by 1
  6. 2026Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitionerscited by 1
  7. 2025Belief Attribution as Mental Explanation: The Role of Accuracy, Informativity, and Causalitycited by 0
  8. 2025What's in the Box? Reasoning about Unseen Objects from Multimodal Cuescited by 0
  9. 2025Reversing the Lens: Using Explainable AI to Understand Human Expertisecited by 0
  10. 2026Predicting Biased Human Decision-Making with Large Language Models in Conversational Settingscited by 0
  11. 2026Emulating Aggregate Human Choice Behavior and Biases with GPT Conversational Agentscited by 0
  12. 2026CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognitioncited by 0
  13. 2026Multi-System Cognitive Steering for Generative Communicationcited by 0