Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigation
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Most Vision-and-Language Navigation (VLN) algorithms are prone to making inaccurate decisions due to their lack of visual common sense and limited reasoning capabilities. To address this issue, we propose a Hierarchical Spatial Proximity Reasoning (HSPR) method. First, we introduce a scene understanding auxiliary task to help the agent build a knowledge base of hierarchical spatial proximity. This task utilizes panoramic views and object features to identify types of nodes and uncover the adjacency relationships between nodes, objects, and between nodes and objects. Second, we propose a multi-step reasoning navigation algorithm based on the hierarchical spatial proximity knowledge base, which continuously plans feasible paths to enhance exploration efficiency. Third, we introduce a residual fusion method to improve navigation decision accuracy. Finally, we validate our approach with experiments on publicly available datasets including REVERIE, SOON, R2R, and R4R. Our code is available at https://github.com/iCityLab/HSPR
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 47 references that are in the map. First posted on arXiv 18 Mar 2024.
Builds on
43 of 47 references.
References · 43
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2015VQA: Visual Question Answeringcited by 2,489
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
31 more
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
- 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
- 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
- 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
- 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
- 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
- 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
- 2018Semi-parametric Topological Memory for Navigationcited by 203
- 2020Neural Topological SLAM for Visual Navigationcited by 193
- 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
- 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
- 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
- 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
- 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
- 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
- 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
- 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
- 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
- 2021Structured Scene Memory for Vision-Language Navigationcited by 95
- 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
- 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
- 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
- 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
- 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
- 2020Auxiliary Tasks Speed Up Learning PointGoal Navigationcited by 52
- 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
- 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
- 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
- 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
- 2023RREx-BoT: Remote Referring Expressions with a Bag of Trickscited by 12
- 2022Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigationcited by 9
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation
- 2025CLASH: Collaborative Large-Small Hierarchical Framework for Continuous Vision-and-Language Navigation
- 2026Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
- 2026SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning
- 2026Think before Go: Hierarchical Reasoning for Image-goal Navigation
- 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
- 2026TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation
- 2025FSR-VLN: Fast and Slow Reasoning for Vision-Language Navigation with Hierarchical Multi-modal Scene Graph