mlsanityData to 14 Oct 2026

Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

arXiv 2412.14171 ↗First posted 18 Dec 2024cs.CVcited by 611

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal Large Language Models (MLLMs) trained on million-scale video datasets also ``think in space'' from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive - though subhuman - visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs' spatial distance ability.

Key facts. Cited by 611 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 79 of its 106 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

79 of 106 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning21 Oct
  2. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  3. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  4. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasoners4 Jun
  5. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  7. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
38 more
  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  13. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  14. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  15. 2022Emergent Abilities of Large Language Modelscited by 2,512
  16. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  17. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  18. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  19. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  20. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  21. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  22. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  23. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  24. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  25. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  26. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  27. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  28. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  29. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  30. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  31. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  32. 2023A Comprehensive Overview of Large Language Modelscited by 648
  33. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  34. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  35. 2024Long Context Transfer from Language to Visioncited by 515
  36. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  37. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  38. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 270; 2026: 341.

611 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  6. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  7. 2025Kimi-VL Technical Reportcited by 388
  8. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  9. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  10. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  11. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  12. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
38 more
  1. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 159
  2. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 138
  3. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  4. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
  5. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  6. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  7. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  8. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 118
  9. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  10. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  11. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  12. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 99
  13. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 93
  14. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  15. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  16. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
  17. 2025Visual Spatial Tuningcited by 89
  18. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 83
  19. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  20. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  21. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  22. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  23. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  24. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  25. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  26. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 66
  27. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  28. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  29. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  30. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  31. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  32. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 59
  33. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 57
  34. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  35. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  36. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  37. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Trainingcited by 51
  38. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51

50 most cited of 611 All 611 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2026How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning
  4. 2026Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
  5. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
  6. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
  7. 2026EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs
  8. 2026Mechanistic Diagnostics of Spatial Lexical Bias in Multimodal Large Language Model Spatial Reasoning