# VLN-Trans: Translator for the Vision and Language Navigation Agent

arXiv: https://arxiv.org/abs/2302.09230 · HTML: https://mlsanity.com/paper/2302.09230/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 41 references that are in the map.
- First posted on arXiv 18 Feb 2023.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation](https://mlsanity.com/paper/1905.12255/) (2019)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation](https://mlsanity.com/paper/1903.01602/) (2019)
- [General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping](https://mlsanity.com/paper/1907.05446/) (2019)
- [Language and Visual Entity Relationship Graph for Agent Navigation](https://mlsanity.com/paper/2010.09304/) (2020)
- [Object-and-Action Aware Model for Visual Language Navigation](https://mlsanity.com/paper/2007.14626/) (2020)
- [Sub-Instruction Aware Vision-and-Language Navigation](https://mlsanity.com/paper/2004.02707/) (2020)
- [Less is More: Generating Grounded Navigation Instructions from Landmarks](https://mlsanity.com/paper/2111.12872/) (2021)
- [Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation](https://mlsanity.com/paper/1906.00347/) (2019)
- [Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigation](https://mlsanity.com/paper/2203.16586/) (2022)
- [Neighbor-view Enhanced Model for Vision and Language Navigation](https://mlsanity.com/paper/2107.07201/) (2021)
- [ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts](https://mlsanity.com/paper/2205.15509/) (2022)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)

## Built on by (most cited first, 25 of 25)

- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [Common Sense Reasoning for Deepfake Detection](https://mlsanity.com/paper/2402.00126/) (2024)
- [Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector](https://mlsanity.com/paper/2503.20188/) (2025)
- [SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models](https://mlsanity.com/paper/2410.03878/) (2024)
- [Controllable Navigation Instruction Generation with Chain of Thought Prompting](https://mlsanity.com/paper/2407.07433/) (2024)
- [Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel](https://mlsanity.com/paper/2412.08467/) (2024)
- [NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM](https://mlsanity.com/paper/2502.11142/) (2025)
- [FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks](https://mlsanity.com/paper/2503.13966/) (2025)
- [Language-guided Hierarchical Fine-grained Image Forgery Detection and Localization](https://mlsanity.com/paper/2410.23556/) (2024)
- [When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning](https://mlsanity.com/paper/2602.08236/) (2026)
- [AeroDuo: Aerial Duo for UAV-based Vision and Language Navigation](https://mlsanity.com/paper/2508.15232/) (2025)
- [NavHint: Vision and Language Navigation Agent with a Hint Generator](https://mlsanity.com/paper/2402.02559/) (2024)
- [ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities](https://mlsanity.com/paper/2410.03907/) (2024)
- [TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation](https://mlsanity.com/paper/2403.08833/) (2024)
- [Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis](https://mlsanity.com/paper/2403.11487/) (2024)
- [NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation](https://mlsanity.com/paper/2510.16457/) (2025)
- [Narrowing the Gap between Vision and Action in Navigation](https://mlsanity.com/paper/2408.10388/) (2024)
- [D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation](https://mlsanity.com/paper/2512.12622/) (2025)
- [On the Holistic Approach for Detecting Human Image Forgery](https://mlsanity.com/paper/2601.04715/) (2026)
- [Trajectory-Diversity-Driven Robust Vision-and-Language Navigation](https://mlsanity.com/paper/2603.15370/) (2026)
- [Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs](https://mlsanity.com/paper/2509.25139/) (2025)
- [VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions](https://mlsanity.com/paper/2604.10533/) (2026)
- [Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?](https://mlsanity.com/paper/2605.30557/) (2026)
- [How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace](https://mlsanity.com/paper/2604.07973/) (2026)
- [Advancing DialNav through Automatic Embodied Dialog Augmentation](https://mlsanity.com/paper/2606.19948/) (2026)

## Abstract (arXiv)

Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent's visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room~(R2R), Room4room~(R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.