# Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

arXiv: https://arxiv.org/abs/2104.09580 · HTML: https://mlsanity.com/paper/2104.09580/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Improving Vision-and-Language Navigation with Image-Text Pairs from the Web](https://mlsanity.com/paper/2004.14973/) (2020)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation](https://mlsanity.com/paper/1905.12255/) (2019)
- [The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation](https://mlsanity.com/paper/1903.01602/) (2019)
- [Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction](https://mlsanity.com/paper/1809.00786/) (2018)
- [Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning](https://mlsanity.com/paper/1909.01871/) (2019)
- [Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation](https://mlsanity.com/paper/1803.07729/) (2018)
- [Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation](https://mlsanity.com/paper/1903.02547/) (2019)
- [Robust Navigation with Language Pretraining and Stochastic Sampling](https://mlsanity.com/paper/1909.02244/) (2019)
- [Object-and-Action Aware Model for Visual Language Navigation](https://mlsanity.com/paper/2007.14626/) (2020)
- [Sub-Instruction Aware Vision-and-Language Navigation](https://mlsanity.com/paper/2004.02707/) (2020)
- [Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation](https://mlsanity.com/paper/1906.00347/) (2019)
- [Active Visual Information Gathering for Vision-Language Navigation](https://mlsanity.com/paper/2007.08037/) (2020)

## Built on by (most cited first, 30 of 31)

- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.19195/) (2023)
- [HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.11591/) (2022)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [Learning Vision-and-Language Navigation from YouTube Videos](https://mlsanity.com/paper/2307.11984/) (2023)
- [Improving Vision-and-Language Navigation by Generating Future-View Image Semantics](https://mlsanity.com/paper/2304.04907/) (2023)
- [ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts](https://mlsanity.com/paper/2205.15509/) (2022)
- [One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones](https://mlsanity.com/paper/2202.07028/) (2022)
- [VLN-Trans: Translator for the Vision and Language Navigation Agent](https://mlsanity.com/paper/2302.09230/) (2023)
- [RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation](https://mlsanity.com/paper/2412.08591/) (2024)
- [Correctable Landmark Discovery via Large Models for Vision-Language Navigation](https://mlsanity.com/paper/2405.18721/) (2024)
- [FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks](https://mlsanity.com/paper/2503.13966/) (2025)
- [Contrastive Instruction-Trajectory Learning for Vision-Language Navigation](https://mlsanity.com/paper/2112.04138/) (2021)
- [SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts](https://mlsanity.com/paper/2412.05552/) (2024)
- [NavHint: Vision and Language Navigation Agent with a Hint Generator](https://mlsanity.com/paper/2402.02559/) (2024)
- [CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations](https://mlsanity.com/paper/2207.02185/) (2022)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [LOViS: Learning Orientation and Visual Signals for Vision and Language Navigation](https://mlsanity.com/paper/2209.12723/) (2022)
- [Rethinking the Spatial Route Prior in Vision-and-Language Navigation](https://mlsanity.com/paper/2110.05728/) (2021)
- [MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigation](https://mlsanity.com/paper/2406.17960/) (2024)
- [NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation](https://mlsanity.com/paper/2510.16457/) (2025)
- [Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding](https://mlsanity.com/paper/2110.04441/) (2021)
- [ULN: Towards Underspecified Vision-and-Language Navigation](https://mlsanity.com/paper/2210.10020/) (2022)
- [Trajectory-Diversity-Driven Robust Vision-and-Language Navigation](https://mlsanity.com/paper/2603.15370/) (2026)
- [Personalized Embodied Navigation for Portable Object Finding](https://mlsanity.com/paper/2403.09905/) (2024)
- [Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2403.03405/) (2024)
- [NAVCON: A Cognitively Inspired and Linguistically Grounded Corpus for Vision and Language Navigation](https://mlsanity.com/paper/2412.13026/) (2024)
- [Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos](https://mlsanity.com/paper/2603.09259/) (2026)

## Abstract (arXiv)

Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations. Code and models: https://github.com/jialuli-luka/SyntaxVLN