mlsanityData to 7 Oct 2026

The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation

arXiv 2104.04167 ↗First posted 9 Apr 2021cs.CL cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) requires an agent to find a path to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas. Most existing methods take the words in the instructions and the discrete views of each panorama as the minimal unit of encoding. However, this requires a model to match different nouns (e.g., TV, table) against the same input view feature. In this work, we propose an object-informed sequential BERT to encode visual perceptions and linguistic instructions at the same fine-grained level, namely objects and words. Our sequential BERT also enables the visual-textual clues to be interpreted in light of the temporal context, which is crucial to multi-round VLN tasks. Additionally, we enable the model to identify the relative direction (e.g., left/right/front/back) of each navigable location and the room type (e.g., bedroom, kitchen) of its current and final navigation goal, as such information is widely mentioned in instructions implying the desired next and final locations. We thus enable the model to know-where the objects lie in the images, and to know-where they stand in the scene. Extensive experiments demonstrate the effectiveness compared against several state-of-the-art methods on three indoor VLN tasks: REVERIE, NDH, and R2R. Project repository: https://github.com/YuankaiQi/ORIST

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 36 of its 42 references that are in the map. First posted on arXiv 9 Apr 2021.

Builds on

36 of 42 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
24 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  2. 2018On Evaluation of Embodied Navigation Agentscited by 526
  3. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  6. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  7. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  8. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  9. 2019Vision-and-Dialog Navigationcited by 246
  10. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  11. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  12. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  13. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  14. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  15. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  16. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  17. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  18. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  19. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  20. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  21. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  22. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37
  23. 2020Vision-Dialog Navigation by Exploring Cross-modal Memorycited by 23
  24. 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 18; 2024: 9; 2025: 10; 2026: 8.

60 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  5. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  6. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  7. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  8. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  9. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  10. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  11. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  12. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
38 more
  1. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  2. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  3. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  4. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
  5. 2022Reinforced Structured State-Evolution for Vision-Language Navigationcited by 36
  6. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  7. 20221st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)cited by 30
  8. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 30
  9. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  10. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  11. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  12. 2023GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigationcited by 24
  13. 2023VLN-PETL: Parameter-Efficient Transfer Learning for Vision-and-Language Navigationcited by 22
  14. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  15. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  16. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  17. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  18. 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 13
  19. 2023RREx-BoT: Remote Referring Expressions with a Bag of Trickscited by 12
  20. 2023Vision and Language Navigation in the Real World via Online Visual Language Mappingcited by 11
  21. 2023Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environmentscited by 9
  22. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  23. 2022FedVLN: Privacy-preserving Federated Vision-and-Language Navigationcited by 8
  24. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  25. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  26. 2024Augmented Commonsense Knowledge for Remote Object Groundingcited by 7
  27. 2022Self-supervised 3D Semantic Representation Learning for Vision-and-Language Navigationcited by 6
  28. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  29. 2022Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTscited by 5
  30. 2022A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cuescited by 5
  31. 2023Actional Atomic-Concept Learning for Demystifying Vision-Language Navigationcited by 4
  32. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  33. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 4
  34. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  35. 2022Local Slot Attention for Vision-and-Language Navigationcited by 3
  36. 2025Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments with a Hierarchical Spatial-Cognition Long-Short Memory Systemcited by 3
  37. 2022Bridging the visual gap in VLN via semantically richer instructionscited by 2
  38. 2024Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIMcited by 2

50 most cited of 60 All 60 (JSON)