mlsanityData to 14 Oct 2026

Mind's Eye: Grounded Language Model Reasoning through Simulation

arXiv 2210.05359 ↗First posted 11 Oct 2022cs.CL cs.AIcited by 84

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Successful and effective communication between humans and AI relies on a shared experience of the world. By training solely on written text, current language models (LMs) miss the grounded experience of humans in the real-world -- their failure to relate language to the physical world causes knowledge to be misrepresented and obvious mistakes in their reasoning. We present Mind's Eye, a paradigm to ground language model reasoning in the physical world. Given a physical reasoning question, we use a computational physics engine (DeepMind's MuJoCo) to simulate the possible outcomes, and then use the simulation results as part of the input, which enables language models to perform reasoning. Experiments on 39 tasks in a physics alignment benchmark demonstrate that Mind's Eye can improve reasoning ability by a large margin (27.9% zero-shot, and 46.0% few-shot absolute accuracy improvement on average). Smaller language models armed with Mind's Eye can obtain similar performance to models that are 100x larger. Finally, we confirm the robustness of Mind's Eye through ablation studies.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 56 of its 85 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

56 of 85 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Language models show human-like content effects on reasoning tasks14 Jul
  2. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoning15 Feb
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  13. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  14. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  15. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  16. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  17. 2022A Generalist Agentcited by 808
  18. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  19. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  20. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  21. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  22. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  23. 2022Kubric: A scalable dataset generatorcited by 381
  24. 2016Solving General Arithmetic Word Problemscited by 360
  25. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  26. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  27. 2022Can language models learn from explanations in context?cited by 249
  28. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  29. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  30. 2022Multi-Game Decision Transformerscited by 214
  31. 2022TALM: Tool Augmented Language Modelscited by 197
  32. 2020How Context Affects Language Models' Factual Predictionscited by 181
  33. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  34. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  35. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  36. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  37. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  38. 2019Environmental drivers of systematicity and generalization in a situated agentcited by 50

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 39; 2024: 19; 2025: 10; 2026: 12.

84 papers.

Sort:

By area:

  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  3. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  4. 2023Cognitive Architectures for Language Agentscited by 429
  5. 2023Augmented Language Models: a Surveycited by 403
  6. 2023Language Models can Solve Computer Taskscited by 372
  7. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  8. 2023Large Language Models as Tool Makerscited by 303
  9. 2023Tool Learning with Foundation Modelscited by 302
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  11. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  12. 2022Reasoning with Language Model Prompting: A Surveycited by 252
38 more
  1. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 168
  2. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  4. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  5. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  6. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  7. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  8. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
  9. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  10. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  11. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  12. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  13. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  14. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  15. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  16. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 70
  17. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  18. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  19. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  20. 2024CRISPR-GPT for Agentic Automation of Gene-editing Experimentscited by 58
  21. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  22. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37
  23. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  24. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  25. 2023Interactive Natural Language Processingcited by 31
  26. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  27. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 30
  28. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 30
  29. 2023Are Large Language Models Temporally Grounded?cited by 27
  30. 2023LLaMA Rider: Spurring Large Language Models to Explore the Open Worldcited by 21
  31. 2026Large Language Model Reasoning Failurescited by 21
  32. 2024A social path to human-like artificial intelligencecited by 20
  33. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructionscited by 19
  34. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 13
  35. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  36. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Modelscited by 11
  37. 2024Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Modelscited by 10
  38. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
  2. 2025Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decomposition
  3. 2025On the Thinking-Language Modeling Gap in Large Language Models
  4. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities
  5. 2023Understanding Social Reasoning in Language Models with Language Models
  6. 2025Language Models Might Not Understand You: Evaluating Theory of Mind via Story Prompting
  7. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  8. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models