mlsanityData to 7 Oct 2026

Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks

arXiv 1911.07883 ↗First posted 18 Nov 2019cs.CVcited by 177

Not on the map. Why?

Abstract (arXiv)

Vision-Language Navigation (VLN) is a task where agents learn to navigate following natural language instructions. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional approaches exploit the vision and language features in cross-modal grounding. However, the VLN task remains challenging, since previous works have neglected the rich semantic information contained in the environment (such as implicit navigation graphs or sub-trajectory semantics). In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information. The auxiliary tasks have four reasoning objectives: explaining the previous actions, estimating the navigation progress, predicting the next orientation, and evaluating the trajectory consistency. As a result, these additional training signals help the agent to acquire knowledge of semantic representations in order to reason about its activity and build a thorough perception of the environment. Our experiments indicate that auxiliary reasoning tasks improve both the performance of the main task and the model generalizability by a large margin. Empirically, we demonstrate that an agent trained with self-supervised auxiliary reasoning tasks substantially outperforms the previous state-of-the-art method, being the best existing approach on the standard benchmark.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 38; 2022: 29; 2023: 38; 2024: 31; 2025: 23; 2026: 16.

177 papers.

Sort:

By area:

  1. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  5. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  7. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  8. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  9. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  10. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  11. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  12. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
38 more
  1. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  2. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  3. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  4. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  5. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  6. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  7. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  8. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  9. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  10. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  11. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  12. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  13. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  14. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  15. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  16. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  17. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  18. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  19. 2023ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidancecited by 70
  20. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  21. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  22. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  23. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  24. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  25. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  26. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  27. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  28. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  29. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  30. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  31. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  32. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  33. 2022Multi-Modal Cross-Domain Alignment Network for Video Moment Retrievalcited by 43
  34. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  35. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  36. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 40
  37. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  38. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39

50 most cited of 177 All 177 (JSON)