mlsanityData to 14 Oct 2026

InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning

arXiv 2609.37187 ↗First posted 29 Sep 2026cs.CV cs.ROcited by 0

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Language-guided navigation requires connecting partial observations to a persistent spatial reference and learning how actions change that representation. We introduce InsightMap, a framework that uses top-down maps as both explicit spatial memory and action-conditioned prediction targets. Historical views are linked to labeled map locations, and a shared multimodal backbone jointly learns navigation action prediction and post-action map generation. Map prediction provides auxiliary training supervision, while navigation inference decodes actions from the observed spatial context. An aligned RGB-D data pipeline supports a common interface for navigation, visual question answering, situated reasoning, and 3D grounding. On the validation-unseen splits of R2R-CE and RxR-CE, InsightMap achieves success rates (SR) of 56.9% and 54.9%, respectively. Adding map-prediction supervision improves R2R-CE SR by 4.3 and success weighted by path length (SPL) by 3.2 percentage points. On static spatial tasks, InsightMap achieves 103.7 CIDEr on ScanQA, 60.1% exact-match accuracy on SQA3D, and 53.1% grounding accuracy at 0.5 IoU on ScanRefer with detected object proposals. On Unitree Go2, it outperforms NaVid and NaVILA in hallway, lab, and office environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 37 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

31 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 20233D-LLM: Injecting the 3D World into Large Language Models24 Jul
  2. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct

Other references · 29

  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  4. 2026World Action Models are Zero-shot Policiescited by 465
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  6. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  7. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  8. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  9. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  10. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  11. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  12. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
17 more
  1. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  2. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  3. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  4. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 156
  5. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  7. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  8. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  9. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  10. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122
  11. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  12. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  13. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 86
  14. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  15. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  16. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  17. 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 60

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
  2. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
  3. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  4. 2026Embodied-Navigator: Point, Think, Memorize, and Align for Efficient Navigation
  5. 2026SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments
  6. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks
  7. 2026Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
  8. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods