mlsanityData to 7 Oct 2026

VLN-Trans: Translator for the Vision and Language Navigation Agent

arXiv 2302.09230 ↗First posted 18 Feb 2023cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent's visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room~(R2R), Room4room~(R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 38 of its 41 references that are in the map. First posted on arXiv 18 Feb 2023.

Builds on

38 of 41 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  12. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
26 more
  1. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  2. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  3. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  4. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  5. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  6. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  7. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  8. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  9. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  10. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  11. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  12. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  13. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  14. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  15. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  16. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  17. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  18. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  19. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  20. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  21. 2020Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rulecited by 22
  22. 2022FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigationcited by 22
  23. 2021Contrastive Instruction-Trajectory Learning for Vision-Language Navigationcited by 16
  24. 2021Towards Navigation by Reasoning over Spatial Configurationscited by 14
  25. 2022LOViS: Learning Orientation and Visual Signals for Vision and Language Navigationcited by 7
  26. 2020From Spatial Relations to Spatial Configurationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 7; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  2. 2024Common Sense Reasoning for Deepfake Detectioncited by 64
  3. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 38
  4. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  5. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  6. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  7. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  8. 2024Language-guided Hierarchical Fine-grained Image Forgery Detection and Localizationcited by 16
  9. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  10. 2025AeroDuo: Aerial Duo for UAV-based Vision and Language Navigationcited by 13
  11. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  12. 2026When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoningcited by 12
13 more
  1. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activitiescited by 9
  2. 2024TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigationcited by 7
  3. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  4. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  5. 2024Narrowing the Gap between Vision and Action in Navigationcited by 5
  6. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  7. 2026On the Holistic Approach for Detecting Human Image Forgerycited by 5
  8. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  9. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMscited by 3
  10. 2026VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructionscited by 2
  11. 2026Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?cited by 2
  12. 2026How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspacecited by 0
  13. 2026Advancing DialNav through Automatic Embodied Dialog Augmentationcited by 0