mlsanityData to 14 Oct 2026

Evaluating Cognitive Maps and Planning in Large Language Models with CogEval

arXiv 2309.15129 ↗First posted 25 Sep 2023cs.AI cs.CL cs.LGcited by 77

Not on the map. Why?

Abstract (arXiv)

Recently an influx of studies claim emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in Large Language Models. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and getting trapped in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 32 of its 69 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

32 of 69 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
20 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  3. 2022Using cognitive psychology to understand GPT-3cited by 398
  4. 2018From Louvain to Leiden: guaranteeing well-connected communitiescited by 385
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  6. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  7. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  8. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  9. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  10. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  11. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  12. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  13. 2019Certified Reinforcement Learning with Logic Guidancecited by 36
  14. 2022Multi-Stage Prompting for Knowledgeable Dialogue Generationcited by 22
  15. 2018Logically-Constrained Neural Fitted Q-Iterationcited by 17
  16. 2023Large language models can segment narrative events similarly to humanscited by 16
  17. 2023GPT is becoming a Turing machine: Here are some ways to program itcited by 16
  18. 2023Imitation versus Innovation: What children can do that large language and language-and-vision models cannot (yet)?cited by 13
  19. 2022Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methodscited by 10
  20. 2022A Rubric for Human-like Agents and NeuroAIcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 40; 2025: 26; 2026: 6.

77 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  2. 2024The pitfalls of next-token predictioncited by 160
  3. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  5. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 102
  6. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  7. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  8. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  9. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  10. 2024Can large language models explore in-context?cited by 76
  11. 2024Building Machines that Learn and Think with Peoplecited by 73
  12. 2023Evaluating Spatial Understanding of Large Language Modelscited by 66
38 more
  1. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 66
  2. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
  3. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  4. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  5. 2024Evaluating Large Language Models as Generative User Simulators for Conversational Recommendationcited by 48
  6. 2024Can Graph Learning Improve Planning in LLM-based Agents?cited by 43
  7. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  8. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
  9. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  10. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  11. 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Modelscited by 30
  12. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  13. 2024Strong and weak alignment of large language models with human valuescited by 27
  14. 2024Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructionscited by 25
  15. 2025Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometricscited by 23
  16. 2024ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Modelscited by 18
  17. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  18. 2023Structured World Representations in Maze-Solving Transformerscited by 16
  19. 2025Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planningcited by 15
  20. 2024Transformers Use Causal World Models in Maze-Solving Taskscited by 13
  21. 2024The Belief State Transformercited by 12
  22. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
  23. 2024Collective Innovation in Groups of Large Language Modelscited by 11
  24. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 11
  25. 2025TALES: Text Adventure Learning Environment Suitecited by 11
  26. 2024What is it for a Machine Learning Model to Have a Capability?cited by 10
  27. 2024Baba Is AI: Break the Rules to Beat the Benchmarkcited by 10
  28. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  29. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  30. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  31. 2023Decoding In-Context Learning: Neuroscience-inspired Analysis of Representations in Large Language Modelscited by 7
  32. 2024How language models extrapolate outside the training data: A case study in Textualized Gridworldcited by 7
  33. 2024On Logical Extrapolation for Mazes with Recurrent and Implicit Networkscited by 7
  34. 2024Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environmentscited by 6
  35. 2024Evaluation of Geographical Distortions in Language Modelscited by 6
  36. 2025Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Predictioncited by 6
  37. 2024Integration of cognitive tasks into artificial general intelligence test for large modelscited by 5
  38. 2025Position: Stop Acting Like Language Model Agents Are Normal Agentscited by 5

50 most cited of 77 All 77 (JSON)