mlsanityData to 7 Oct 2026

Target-Driven Structured Transformer Planner for Vision-Language Navigation

arXiv 2207.11201 ↗First posted 19 Jul 2022cs.CV cs.AI cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

Vision-language navigation is the task of directing an embodied agent to navigate in 3D scenes with natural language instructions. For the agent, inferring the long-term navigation target from visual-linguistic clues is crucial for reliable path planning, which, however, has rarely been studied before in literature. In this article, we propose a Target-Driven Structured Transformer Planner (TD-STP) for long-horizon goal-guided and room layout-aware navigation. Specifically, we devise an Imaginary Scene Tokenization mechanism for explicit estimation of the long-term target (even located in unexplored environments). In addition, we design a Structured Transformer Planner which elegantly incorporates the explored room layout into a neural attention architecture for structured and global planning. Experimental results demonstrate that our TD-STP substantially improves previous best methods' success rate by 2% and 5% on the test set of R2R and REVERIE benchmarks, respectively. Our code is available at https://github.com/YushengZhao/TD-STP .

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 60 of its 65 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

60 of 65 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
38 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  3. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  6. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  7. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  8. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  9. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 257
  10. 2019Vision-and-Dialog Navigationcited by 246
  11. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  12. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  13. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  14. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  15. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  16. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  17. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  18. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  19. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  20. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  21. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  22. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  23. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 109
  24. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  25. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  26. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  27. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  28. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  29. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  30. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  31. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  32. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  33. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  34. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  35. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  36. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  37. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  38. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 17; 2024: 17; 2025: 13; 2026: 9.

58 papers.

Sort:

By area:

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  2. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  3. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  4. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  5. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  6. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  7. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  8. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  9. 2023A Comprehensive Survey on Deep Graph Representation Learningcited by 72
  10. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  11. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  12. 2024Vision-and-Language Navigation via Causal Learningcited by 51
38 more
  1. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  2. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  3. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  4. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  5. 2022Towards Versatile Embodied Navigationcited by 26
  6. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  7. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  8. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  9. 2023Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigationcited by 18
  10. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  11. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  12. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  13. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigationcited by 15
  14. 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Taskcited by 14
  15. 2025AeroDuo: Aerial Duo for UAV-based Vision and Language Navigationcited by 13
  16. 2023RREx-BoT: Remote Referring Expressions with a Bag of Trickscited by 12
  17. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  18. 2023Graph Neural Networks in Vision-Language Image Understanding: A Surveycited by 10
  19. 2023ESceme: Vision-and-Language Navigation with Episodic Scene Memorycited by 9
  20. 2023PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigationcited by 8
  21. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  22. 2024Augmented Commonsense Knowledge for Remote Object Groundingcited by 7
  23. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  24. 2024Vision-Language Navigation with Continual Learningcited by 6
  25. 2024MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillationcited by 6
  26. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  27. 2025Attention Bootstrapping for Multi-Modal Test-Time Adaptationcited by 5
  28. 2025History-Enhanced Two-Stage Transformer for Aerial Vision-and-Language Navigationcited by 5
  29. 2025MSNav: Zero-Shot Vision-and-Language Navigation with Dynamic Memory and LLM Spatial Reasoningcited by 4
  30. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  31. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  32. 2023What Is Near?: Room Locality Learning for Enhanced Robot Vision-Language-Navigation in Indoor Living Environmentscited by 2
  33. 2024COOL: A Conjoint Perspective on Spatio-Temporal Graph Neural Network for Traffic Forecastingcited by 2
  34. 2024World-Consistent Data Generation for Vision-and-Language Navigationcited by 2
  35. 2026TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigationcited by 2
  36. 2026Does Peer Observation Help? Vision-Sharing Collaboration for Vision-Language Navigationcited by 2
  37. 2024Graph Neural Networks in Intelligent Transportation Systems: Advances, Applications and Trendscited by 1
  38. 2025Efficient and Generalizable Environmental Understanding for Visual Navigationcited by 1

50 most cited of 58 All 58 (JSON)