mlsanityData to 7 Oct 2026

From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D

arXiv 2503.22976 ↗First posted 29 Mar 2025cs.CVcited by 127

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advances in LVLMs have improved vision-language understanding, but they still struggle with spatial perception, limiting their ability to reason about complex 3D scenes. Unlike previous approaches that incorporate 3D representations into models to improve spatial understanding, we aim to unlock the potential of VLMs by leveraging spatially relevant image data. To this end, we introduce a novel 2D spatial data generation and annotation pipeline built upon scene data with 3D ground-truth. This pipeline enables the creation of a diverse set of spatial tasks, ranging from basic perception tasks to more complex reasoning tasks. Leveraging this pipeline, we construct SPAR-7M, a large-scale dataset generated from thousands of scenes across multiple public datasets. In addition, we introduce SPAR-Bench, a benchmark designed to offer a more comprehensive evaluation of spatial capabilities compared to existing spatial benchmarks, supporting both single-view and multi-view inputs. Training on both SPAR-7M and large-scale 2D datasets enables our models to achieve state-of-the-art performance on 2D spatial benchmarks. Further fine-tuning on 3D task-specific datasets yields competitive results, underscoring the effectiveness of our dataset in enhancing spatial reasoning.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 65 references that are in the map. First posted on arXiv 29 Mar 2025.

Builds on

51 of 65 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Hypo3D: Exploring Hypothetical Reasoning in 3D2 Feb
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  3. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  4. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
  5. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  8. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2019Towards VQA Models That Can Readcited by 1,696
31 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  8. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  9. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  10. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  11. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  12. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  13. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  14. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  15. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  16. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  17. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  18. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  19. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  20. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  21. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  22. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  23. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  24. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  25. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  26. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  27. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  28. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  29. 2024ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Modelscited by 54
  30. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  31. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 39; 2026: 88.

127 papers.

Sort:

By area:

  1. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  2. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  3. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2025Visual Spatial Tuningcited by 85
  6. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  7. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 58
  8. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  9. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  10. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  11. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  12. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
38 more
  1. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  2. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  3. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 20
  4. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  5. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  6. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  7. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 14
  8. 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoningcited by 13
  9. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 12
  10. 2025VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimizationcited by 11
  11. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  12. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  13. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoningcited by 11
  14. 2026ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoningcited by 11
  15. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 10
  16. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 9
  17. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  18. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 8
  19. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  20. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 7
  21. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  22. 2026On the Generalization Capacities of MLLMs for Spatial Intelligencecited by 7
  23. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
  24. 2025Video Spatial Reasoning with Object-Centric 3D Rolloutcited by 6
  25. 2026Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoningcited by 6
  26. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 6
  27. 2026GEM: Generative Supervision Helps Embodied Intelligencecited by 6
  28. 2026From Pixels to Words -- Towards Native One-Vision Models at Scalecited by 6
  29. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 5
  30. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 5
  31. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  32. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  33. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  34. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibilitycited by 4
  35. 2026Make Geometry Matter for Spatial Reasoningcited by 4
  36. 2025MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Imagescited by 3
  37. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  38. 2025Scaling Spatial Reasoning in MLLMs through Programmatic Data Synthesiscited by 3

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models
  2. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
  3. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
  4. 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generation
  5. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
  6. 2025LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
  7. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  8. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data