mlsanityData to 7 Oct 2026

Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation

arXiv 2202.11742 ↗First posted 23 Feb 2022cs.CVcited by 187

Not on the map. Why?

Abstract (arXiv)

Following language instructions to navigate in unseen environments is a challenging problem for autonomous embodied agents. The agent not only needs to ground languages in visual scenes, but also should explore the environment to reach its target. In this work, we propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation (VLN) benchmarks REVERIE and SOON. It also improves the success rate on the fine-grained VLN benchmark R2R.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Builds on 48 of its 59 references that are in the map. First posted on arXiv 23 Feb 2022.

Builds on

48 of 59 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
36 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  3. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2017Embodied Question Answeringcited by 335
  6. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  7. 2020Learning to Explore using Active Neural SLAMcited by 308
  8. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  9. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  10. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  11. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  12. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  13. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  14. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  15. 2018Semi-parametric Topological Memory for Navigationcited by 203
  16. 2020Neural Topological SLAM for Visual Navigationcited by 193
  17. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  18. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  19. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  20. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  21. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  22. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  23. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  24. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  25. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  26. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  27. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  28. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  29. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  30. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  31. 2019Multi-Target Embodied Question Answeringcited by 74
  32. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  33. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  34. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  35. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
  36. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 38; 2024: 35; 2025: 46; 2026: 56.

187 papers.

Sort:

By area:

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  2. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  3. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  4. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  5. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  6. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  7. 2023AerialVLN: Vision-and-Language Navigation for UAVscited by 117
  8. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  9. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  10. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  11. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  12. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
38 more
  1. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  4. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  5. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  6. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  7. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  8. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  9. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  10. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  11. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  12. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  13. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  14. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  15. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  16. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  17. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  18. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  19. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  20. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  21. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  22. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  23. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  24. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
  25. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  26. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  27. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  28. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  29. 2025UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learningcited by 30
  30. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  31. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  32. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  33. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  34. 2022Multi-Object Navigation with dynamically learned neural implicit representationscited by 21
  35. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  36. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  37. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  38. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19

50 most cited of 187 All 187 (JSON)