mlsanityData to 7 Oct 2026

Structured Scene Memory for Vision-Language Navigation

arXiv 2103.03454 ↗First posted 5 Mar 2021cs.CV cs.AIcited by 95

Not on the map. Why?

Abstract (arXiv)

Recently, numerous algorithms have been developed to tackle the problem of vision-language navigation (VLN), i.e., entailing an agent to navigate 3D environments through following linguistic instructions. However, current VLN agents simply store their past experiences/observations as latent states in recurrent networks, failing to capture environment layouts and make long-term planning. To address these limitations, we propose a crucial architecture, called Structured Scene Memory (SSM). It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment. SSM has a collect-read controller that adaptively collects information for supporting current decision making and mimics iterative algorithms for long-range reasoning. As SSM provides a complete action space, i.e., all the navigable places on the map, a frontier-exploration based navigation decision making strategy is introduced to enable efficient and global planning. Experiment results on two VLN datasets (i.e., R2R and R4R) show that our method achieves state-of-the-art performance on several metrics.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 5 Mar 2021.

Builds on

40 of 57 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2014Neural Turing Machinescited by 630
  6. 2018On Evaluation of Embodied Navigation Agentscited by 526
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  8. 2014Memory Networkscited by 354
  9. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2020Learning to Explore using Active Neural SLAMcited by 308
  12. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
28 more
  1. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  2. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  3. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  4. 2018Semi-parametric Topological Memory for Navigationcited by 203
  5. 2020Neural Topological SLAM for Visual Navigationcited by 193
  6. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  7. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  8. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  9. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  10. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  11. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  12. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  13. 2017Neural Map: Structured Memory for Deep Reinforcement Learningcited by 88
  14. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  15. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  16. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  17. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  18. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  19. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  20. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  21. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 54
  22. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  23. 2019Bayesian Relational Memory for Semantic Visual Navigationcited by 48
  24. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
  25. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  26. 2019A Behavioral Approach to Visual Navigation with Graph Localization Networkscited by 40
  27. 2018Gated Path Planning Networkscited by 31
  28. 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 19; 2023: 23; 2024: 18; 2025: 14; 2026: 14.

95 papers.

Sort:

By area:

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  2. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  3. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  4. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  5. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  6. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  7. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  8. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  9. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  10. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  11. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  12. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
38 more
  1. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  4. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  5. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  6. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  7. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  8. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  9. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  10. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  11. 2021Three Steps to Multimodal Trajectory Prediction: Modality Clustering, Classification and Synthesiscited by 46
  12. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  13. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  14. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  15. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
  16. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  17. 2022Reinforced Structured State-Evolution for Vision-Language Navigationcited by 36
  18. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  19. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  20. 2022Towards Versatile Embodied Navigationcited by 26
  21. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  22. 2022FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigationcited by 22
  23. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  24. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  25. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  26. 2021Collaborative Visual Navigationcited by 18
  27. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  28. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  29. 2023Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigationcited by 18
  30. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  31. 2025Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environmentscited by 13
  32. 2023RREx-BoT: Remote Referring Expressions with a Bag of Trickscited by 12
  33. 2024NavHint: Vision and Language Navigation Agent with a Hint Generatorcited by 12
  34. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 11
  35. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 11
  36. 2023Graph Neural Networks in Vision-Language Image Understanding: A Surveycited by 10
  37. 2023Graph based Environment Representation for Vision-and-Language Navigation in Continuous Environmentscited by 9
  38. 2023ESceme: Vision-and-Language Navigation with Episodic Scene Memorycited by 9

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MSNav: Zero-Shot Vision-and-Language Navigation with Dynamic Memory and LLM Spatial Reasoning
  2. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains
  3. 2026StageVLN: Spatial and Trajectory Auxiliary Guidance for Efficient Vision-Language Navigation
  4. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigation
  5. 2025Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments with a Hierarchical Spatial-Cognition Long-Short Memory System
  6. 2026HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
  7. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
  8. 2026Embodied-Navigator: Point, Think, Memorize, and Align for Efficient Navigation