mlsanityData to 7 Oct 2026

RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation

arXiv 2412.08591 ↗First posted 11 Dec 2024cs.CV cs.AI cs.ROcited by 23

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) suffers from the limited diversity and scale of training data, primarily constrained by the manual curation of existing simulators. To address this, we introduce RoomTour3D, a video-instruction dataset derived from web-based room tour videos that capture real-world indoor spaces and human walking demonstrations. Unlike existing VLN datasets, RoomTour3D leverages the scale and diversity of online videos to generate open-ended human walking trajectories and open-world navigable instructions. To compensate for the lack of navigation data in online videos, we perform 3D reconstruction and obtain 3D trajectories of walking paths augmented with additional information on the room types, object locations and 3D shape of surrounding scenes. Our dataset includes ~100K open-ended description-enriched trajectories with ~200K instructions, and 17K action-enriched trajectories from 1847 room tour environments. We demonstrate experimentally that RoomTour3D enables significant improvements across multiple VLN tasks including CVDN, SOON, R2R, and REVERIE. Moreover, RoomTour3D facilitates the development of trainable zero-shot VLN agents, showcasing the potential and challenges of advancing towards open-world navigation.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 48 of its 63 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

48 of 63 references.

References · 48

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  5. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  7. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  8. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  9. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2017Embodied Question Answeringcited by 335
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
36 more
  1. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  2. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  5. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  6. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  7. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  8. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  9. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  10. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  11. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  12. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  13. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  14. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  15. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  16. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  17. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  18. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  19. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  20. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  21. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  22. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  23. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  24. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54
  25. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  26. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  27. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  28. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  29. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  30. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  31. 2022Visual-Language Navigation Pretraining via Prompt-based Environmental Self-explorationcited by 25
  32. 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22
  33. 2023VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigationcited by 22
  34. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  35. 2024OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representationcited by 17
  36. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  2. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  3. 2026Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generationcited by 12
  4. 2025IndoorUAV: Benchmarking Vision-Language UAV Navigation in Continuous Indoor Environmentscited by 9
  5. 2025PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assemblycited by 7
  6. 2026SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenancecited by 6
  7. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  8. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  9. 2026OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMscited by 4
  10. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  11. 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
  12. 2026Lifelong Embodied Navigation Learningcited by 3
11 more
  1. 2025Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigationcited by 2
  2. 2026LIME: Learning Intent-aware Camera Motion from Egocentric Videocited by 2
  3. 2025Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusioncited by 1
  4. 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigationcited by 0
  5. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0
  6. 2026Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigationcited by 0
  7. 2026Lifting Unlabeled Internet-level Data for 3D Scene Understandingcited by 0
  8. 2026CoFL-S: Spatially Queryable Sector Flow Fields for Local Language-Conditioned Navigationcited by 0
  9. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  10. 2026ChainSpace: A Chained-Reasoning Paradigm for Spatial Intelligencecited by 0
  11. 2026Agentic Video Understanding: A Surveycited by 0