mlsanityData to 7 Oct 2026

ESceme: Vision-and-Language Navigation with Episodic Scene Memory

arXiv 2303.01032 ↗First posted 2 Mar 2023cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Vision-and-language navigation (VLN) simulates a visual agent that follows natural-language navigation instructions in real-world scenes. Existing approaches have made enormous progress in navigation in new environments, such as beam search, pre-exploration, and dynamic or hierarchical history encoding. To balance generalization and efficiency, we resort to memorizing visited scenarios apart from the ongoing route while navigating. In this work, we introduce a mechanism of Episodic Scene memory (ESceme) for VLN that wakes an agent's memories of past visits when it enters the current scene. The episodic scene memory allows the agent to envision a bigger picture of the next prediction. This way, the agent learns to utilize dynamically updated information instead of merely adapting to the current observations. We provide a simple yet effective implementation of ESceme by enhancing the accessible views at each location and progressively completing the memory while navigating. We verify the superiority of ESceme on short-horizon (R2R), long-horizon (R4R), and vision-and-dialog (CVDN) VLN tasks. Our ESceme also wins first place on the CVDN leaderboard. Code is available: urlhttps://github.com/qizhust/esceme.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 52 of its 68 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

52 of 68 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2020Benchmarking Graph Neural Networkscited by 672
  8. 2018On Evaluation of Embodied Navigation Agentscited by 526
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  10. 2019Provably Powerful Graph Networkscited by 385
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
38 more
  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  2. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2019Vision-and-Dialog Navigationcited by 246
  5. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  6. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  7. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  8. 2020Neural Topological SLAM for Visual Navigationcited by 193
  9. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  10. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  11. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  12. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  13. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  14. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  15. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  16. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  17. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  18. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  19. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  20. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  21. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  22. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  23. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  24. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  25. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  26. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  27. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  28. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  29. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  30. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  31. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  32. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  33. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  34. 2022Reinforced Structured State-Evolution for Vision-Language Navigationcited by 36
  35. 2022Iterative Vision-and-Language Navigationcited by 34
  36. 2022Episodic Memory Question Answeringcited by 32
  37. 2020Vision-Dialog Navigation by Exploring Cross-modal Memorycited by 23
  38. 2021Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigationcited by 20

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 4.

9 papers.

  1. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  2. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  3. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigationcited by 15
  4. 2025LongFly: Long-Horizon UAV Vision-and-Language Navigation with Spatiotemporal Context Integrationcited by 12
  5. 2026MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon taskscited by 8
  6. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3
  7. 2026Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigationcited by 0
  8. 2026Robo-Cortex: A Self-Evolving Embodied Agent via Dual-Grain Cognitive Memory and Autonomous Knowledge Inductioncited by 0
  9. 2026VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memorycited by 0