mlsanityData to 14 Oct 2026

What Is Near?: Room Locality Learning for Enhanced Robot Vision-Language-Navigation in Indoor Living Environments

arXiv 2309.05036 ↗First posted 10 Sep 2023cs.RO cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Humans use their knowledge of common house layouts obtained from previous experiences to predict nearby rooms while navigating in new environments. This greatly helps them navigate previously unseen environments and locate their target room. To provide layout prior knowledge to navigational agents based on common human living spaces, we propose WIN (textitWhat textitIs textitNear), a commonsense learning model for Vision Language Navigation (VLN) tasks. VLN requires an agent to traverse indoor environments based on descriptive navigational instructions. Unlike existing layout learning works, WIN predicts the local neighborhood map based on prior knowledge of living spaces and current observation, operating on an imagined global map of the entire environment. The model infers neighborhood regions based on visual cues of current observations, navigational history, and layout common sense. We show that local-global planning based on locality knowledge and predicting the indoor layout allows the agent to efficiently select the appropriate action. Specifically, we devised a cross-modal transformer that utilizes this locality prior for decision-making in addition to visual inputs and instructions. Experimental results show that locality learning using WIN provides better generalizability compared to classical VLN agents in unseen environments. Our model performs favorably on standard VLN metrics, with Success Rate 68% and Success weighted by Path Length 63% in unseen environments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 37 references that are in the map. First posted on arXiv 10 Sep 2023.

Builds on

34 of 37 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  7. 2018On Evaluation of Embodied Navigation Agentscited by 547
  8. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  9. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  10. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  11. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  12. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
22 more
  1. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  2. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  3. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  4. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  5. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  6. 2018Visual Semantic Navigation using Scene Priorscited by 128
  7. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 116
  8. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 106
  9. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  10. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  11. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  12. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 64
  13. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 58
  14. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  15. 2019CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysiscited by 50
  16. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
  17. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 44
  18. 2019Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigationcited by 40
  19. 2020Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigationcited by 34
  20. 2020Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigationcited by 22
  21. 2019Multimodal Attention Networks for Low-Level Vision-and-Language Navigationcited by 17
  22. 2019Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularizationcited by 7

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  2. 2024Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generationcited by 3