mlsanityData to 14 Oct 2026

Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks

arXiv 2205.05718 ↗First posted 11 May 2022cs.CL cs.AI cs.LGcited by 56

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Human language offers a powerful window into our thoughts -- we tell stories, give explanations, and express our beliefs and goals through words. Abundant evidence also suggests that language plays a developmental role in structuring our learning. Here, we ask: how much of human-like thinking can be captured by learning statistical patterns in language alone? We first contribute a new challenge benchmark for comparing humans and distributional large language models (LLMs). Our benchmark contains two problem-solving domains (planning and explanation generation) and is designed to require generalization to new, out-of-distribution problems expressed in language. We find that humans are far more robust than LLMs on this benchmark. Next, we propose a hybrid Parse-and-Solve model, which augments distributional LLMs with a structured symbolic reasoning module. We find that this model shows more robust adaptation to out-of-distribution planning problems, demonstrating the promise of hybrid AI models for more human-like reasoning.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 18 references that are in the map. First posted on arXiv 11 May 2022.

Builds on

10 of 18 references.

References · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  7. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  8. 2021Skill Induction and Planning with Latent Languagecited by 99
  9. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  10. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 18; 2024: 18; 2025: 13; 2026: 4.

56 papers.

Sort:

By area:

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2022Using cognitive psychology to understand GPT-3cited by 398
  3. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  4. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  5. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  6. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  7. 2022Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Proverscited by 98
  8. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  9. 2025A Survey of AI Agent Protocolscited by 81
  10. 2023Eight Things to Know about Large Language Modelscited by 73
  11. 2024CogBench: a large language model walks into a psychology labcited by 63
  12. 2024Can Language Models Learn to Skip Steps?cited by 59
38 more
  1. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 49
  2. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  3. 2024A Scalable Communication Protocol for Networks of Large Language Modelscited by 45
  4. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  5. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  6. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 34
  7. 2023Dissociating language and thought in large language modelscited by 29
  8. 2024AI for social science and social science of AI: A Surveycited by 29
  9. 2025On Benchmarking Human-Like Intelligence in Machinescited by 25
  10. 2024A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddingscited by 24
  11. 2023The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputscited by 23
  12. 2024NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptionscited by 22
  13. 2023Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understandingcited by 20
  14. 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 16
  15. 2023Evaluating statistical language models as pragmatic reasonerscited by 15
  16. 2024PROC2PDDL: Open-Domain Planning Representations from Textscited by 15
  17. 2024PDDLEGO: Iterative Planning in Textual Environmentscited by 14
  18. 2025Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Modelscited by 14
  19. 2022Human-in-the-Loop Mixupcited by 8
  20. 2023Towards a Psychological Generalist AI: A Survey of Current Applications of Large Language Models and Future Prospectscited by 8
  21. 2024Can Large Language Models Act as Symbolic Reasoners?cited by 8
  22. 2025Language Models For Generalised PDDL Planning: Synthesising Sound and Programmatic Policiescited by 7
  23. 2024Consolidating Trees of Robotic Plans Generated Using Large Language Models to Improve Reliabilitycited by 6
  24. 2024Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilitiescited by 6
  25. 2024Autonomous Prompt Engineering in Large Language Modelscited by 5
  26. 2025Anticipate & Act : Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environmentscited by 5
  27. 2023Understanding Natural Language Understanding Systems. A Critical Analysiscited by 4
  28. 2023PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planningcited by 4
  29. 2024Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Modelscited by 4
  30. 2024People use fast, goal-directed simulation to reason about novel gamescited by 4
  31. 2025Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Modelscited by 4
  32. 2025Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratingscited by 3
  33. 2025Distinguishing AI-Generated and Human-Written Text Through Psycholinguistic Analysiscited by 3
  34. 2025THiNK: Can Large Language Models Think-aloud?cited by 3
  35. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 2
  36. 2023A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoningcited by 1
  37. 2023Human-Centered Planningcited by 1
  38. 2025Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answeringcited by 1

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models
  2. 2023Studying and improving reasoning in humans and machines
  3. 2025Reasoning Capabilities of Large Language Models on Dynamic Tasks
  4. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
  5. 2025The Philosophical Foundations of Growing AI Like A Child
  6. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  7. 2024Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models
  8. 2025On the Thinking-Language Modeling Gap in Large Language Models