mlsanityData to 14 Oct 2026

SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models

arXiv 2412.07755 ↗First posted 10 Dec 2024cs.CV cs.AI cs.GRcited by 114

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space, i.e., reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset utilizing 3D simulators, comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11% and a LLaVA-Video-7B model by an average 8% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos -- even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 101 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

43 of 101 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  2. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  3. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
  4. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  5. 2024Language-Image Models with 3D Understanding6 May
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  7. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  9. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
22 more
  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  8. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  9. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  10. 2022Visual Spatial Reasoningcited by 293
  11. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  12. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  13. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  14. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  15. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  16. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  17. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  18. 2019SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognitioncited by 55
  19. 20213D Question Answeringcited by 41
  20. 2021Contrastive Syn-to-Real Generalizationcited by 32
  21. 2023From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Spurious Correlations in Image Recognitioncited by 12
  22. 20212.5D Visual Relationship Detectioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 63.

114 papers.

Sort:

By area:

  1. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  2. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  3. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
  4. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 150
  5. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  6. 2025Visual Spatial Tuningcited by 89
  7. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 83
  8. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  9. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  10. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  11. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  12. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
38 more
  1. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 50
  2. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  3. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  4. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 42
  5. 2025Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Modelscited by 38
  6. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  7. 2025Reinforced Visual Perception with Toolscited by 38
  8. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  9. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  10. 20253D Aware Region Prompted Vision Language Modelcited by 33
  11. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  12. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 31
  13. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 29
  14. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  15. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  16. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
  17. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  18. 2025Latent Implicit Visual Reasoningcited by 18
  19. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  20. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 17
  21. 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 17
  22. 2025M2-Reasoning: Empowering MLLMs with Unified General and Spatial Reasoningcited by 16
  23. 2026Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoningcited by 16
  24. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  25. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoningcited by 15
  26. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 14
  27. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  28. 2025How Much 3D Do Video Foundation Models Encode?cited by 11
  29. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  30. 2025Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigationcited by 10
  31. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
  32. 2025Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learningcited by 9
  33. 2025Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Modelscited by 9
  34. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  35. 2025Mind to Hand: Purposeful Robotic Control via Embodied Reasoningcited by 8
  36. 2026Make Geometry Matter for Spatial Reasoningcited by 8
  37. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agentscited by 8
  38. 2025Abstract 3D Perception for Spatial Intelligence in Vision-Language Modelscited by 7

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations
  2. 2026EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs
  3. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
  4. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?
  5. 2025Bridging the Dynamic Perception Gap: Training-Free Draft Chain-of-Thought for Dynamic Multimodal Spatial Reasoning
  6. 2026Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
  7. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  8. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition