mlsanityData to 7 Oct 2026

Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

arXiv 2104.09580 ↗First posted 19 Apr 2021cs.CL cs.AI cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations. Code and models: https://github.com/jialuli-luka/SyntaxVLN

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

39 of 51 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  6. 2018On Evaluation of Embodied Navigation Agentscited by 526
  7. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  8. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  9. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  10. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
27 more
  1. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  2. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  3. 2016Enhanced LSTM for Natural Language Inferencecited by 199
  4. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  5. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  6. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  7. 2016End-to-End Relation Extraction using LSTMs on Sequences and Tree Structurescited by 161
  8. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  9. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  10. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  11. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  12. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  13. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  14. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  15. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  16. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  17. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  18. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  19. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  20. 2017Learning to Compose Task-Specific Tree Structurescited by 48
  21. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
  22. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37
  23. 2016Tree-to-Sequence Attentional Neural Machine Translationcited by 28
  24. 2020Vision-Dialog Navigation by Exploring Cross-modal Memorycited by 23
  25. 2020Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigationcited by 20
  26. 2019Multi-modal Discriminative Model for Vision-and-Language Navigationcited by 17
  27. 2020Multi-View Learning for Vision-and-Language Navigationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 7; 2023: 4; 2024: 10; 2025: 3; 2026: 3.

31 papers.

Sort:

By area:

  1. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  2. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  3. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  4. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  5. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  6. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  7. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  8. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  9. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  10. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  11. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  12. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
19 more
  1. 2021Contrastive Instruction-Trajectory Learning for Vision-Language Navigationcited by 16
  2. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  3. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  4. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  5. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  6. 2022CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representationscited by 10
  7. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  8. 2022LOViS: Learning Orientation and Visual Signals for Vision and Language Navigationcited by 7
  9. 2021Rethinking the Spatial Route Prior in Vision-and-Language Navigationcited by 6
  10. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  11. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  12. 2021Natural Language for Human-Robot Collaboration: Problems Beyond Language Groundingcited by 5
  13. 2022ULN: Towards Underspecified Vision-and-Language Navigationcited by 5
  14. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  15. 2024Personalized Embodied Navigation for Portable Object Findingcited by 3
  16. 2024Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigationcited by 0
  17. 2024NAVCON: A Cognitively Inspired and Linguistically Grounded Corpus for Vision and Language Navigationcited by 0
  18. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0
  19. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigationcited by 0