mlsanityData to 7 Oct 2026

MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation

arXiv 2303.01396 ↗First posted 2 Mar 2023cs.CV cs.CL cs.MMcited by 11

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) aims to develop intelligent agents to navigate in unseen environments only through language and vision supervision. In the recently proposed continuous settings (continuous VLN), the agent must act in a free 3D space and faces tougher challenges like real-time execution, complex instruction understanding, and long action sequence prediction. For a better performance in continuous VLN, we design a multi-level instruction understanding procedure and propose a novel model, Multi-Level Attention Network (MLANet). The first step of MLANet is to generate sub-instructions efficiently. We design a Fast Sub-instruction Algorithm (FSA) to segment the raw instruction into sub-instructions and generate a new sub-instruction dataset named ``FSASub". FSA is annotation-free and faster than the current method by 70 times, thus fitting the real-time requirement in continuous VLN. To solve the complex instruction understanding problem, MLANet needs a global perception of the instruction and observations. We propose a Multi-Level Attention (MLA) module to fuse vision, low-level semantics, and high-level semantics, which produce features containing a dynamic and global comprehension of the task. MLA also mitigates the adverse effects of noise words, thus ensuring a robust understanding of the instruction. To correctly predict actions in long trajectories, MLANet needs to focus on what sub-instruction is being executed every step. We propose a Peak Attention Loss (PAL) to improve the flexible and adaptive selection of the current sub-instruction. PAL benefits the navigation agent by concentrating its attention on the local information, thus helping the agent predict the most appropriate actions. We train and test MLANet in the standard benchmark. Experiment results show MLANet outperforms baselines by a significant margin.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

39 of 56 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2018On Evaluation of Embodied Navigation Agentscited by 526
  10. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
27 more
  1. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  2. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  3. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  4. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  5. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  6. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  7. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  8. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  9. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  10. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  11. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  12. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  13. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  14. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  15. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  16. 2021Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environmentscited by 87
  17. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  18. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  19. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  20. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  21. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  22. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  23. 2020Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Surveycited by 48
  24. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  25. 2020Perception and Navigation in Autonomous Systems in the Era of Learning: A Surveycited by 30
  26. 2021Towards Navigation by Reasoning over Spatial Configurationscited by 14
  27. 2020Multi-View Learning for Vision-and-Language Navigationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 1; 2026: 3.

11 papers.

  1. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  2. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  3. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  4. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  5. 2023PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigationcited by 8
  6. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  7. 2026P2DNav: Panorama-to-Downview Reasoning for Zero-shot Vision-and-Language Navigationcited by 2
  8. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  9. 2024Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigationcited by 0
  10. 2025SEMNAV: Enhancing Visual Semantic Navigation in Robotics through Semantic Segmentationcited by 0
  11. 2026Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videoscited by 0