mlsanityData to 14 Oct 2026

SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation

arXiv 2110.14143 ↗First posted 27 Oct 2021cs.CVcited by 53

Not on the map. Why?

Abstract (arXiv)

Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining (i.e., learning the alignment between images and text from large-scale web data) to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1.8% absolute in SPL on R2R and 3.7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

29 of 36 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
17 more
  1. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  2. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  3. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  4. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  5. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  6. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  7. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  8. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  9. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  10. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  11. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 105
  12. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  13. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  14. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  15. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 55
  16. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
  17. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 11; 2024: 18; 2025: 6; 2026: 9.

53 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 400
  2. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  3. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  4. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  5. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  6. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  7. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  8. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 72
  9. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  10. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  11. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  12. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
38 more
  1. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  2. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  3. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
  4. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
  5. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  6. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 39
  7. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 27
  8. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 25
  9. 2024Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigationcited by 24
  10. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 21
  11. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  12. 2022Unbiased Directed Object Attention Graph for Object Navigationcited by 18
  13. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  14. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 17
  15. 2023Vision and Language Navigation in the Real World via Online Visual Language Mappingcited by 11
  16. 2023Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environmentscited by 9
  17. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  18. 2023Behavioral Analysis of Vision-and-Language Navigation Agentscited by 8
  19. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  20. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  21. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  22. 2022ULN: Towards Underspecified Vision-and-Language Navigationcited by 5
  23. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 5
  24. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 5
  25. 2023Actional Atomic-Concept Learning for Demystifying Vision-Language Navigationcited by 4
  26. 2026Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigationcited by 4
  27. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  28. 2024Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generationcited by 3
  29. 2025TRAVEL: Training-Free Retrieval and Alignment for Vision-and-Language Navigationcited by 3
  30. 2024Can ChatGPT assist visually impaired people with micro-navigation?cited by 2
  31. 2024Causality-Aware Transformer Networks for Robotic Navigationcited by 2
  32. 2025Harnessing Input-Adaptive Inference for Efficient VLNcited by 1
  33. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  34. 2026Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memorycited by 1
  35. 2024Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigationcited by 0
  36. 2024Temporal-Spatial Object Relations Modeling for Vision-and-Language Navigationcited by 0
  37. 2025TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Makingcited by 0
  38. 2026VLN-Pilot: Large Vision-Language Model as an Autonomous Indoor Drone Operatorcited by 0

50 most cited of 53 All 53 (JSON)