mlsanityData to 7 Oct 2026

Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigation

arXiv 2412.01857 ↗First posted 30 Nov 2024cs.CV cs.LG cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Humans navigate unfamiliar environments using episodic simulation and episodic memory, which facilitate a deeper understanding of the complex relationships between environments and objects. Developing an imaginative memory system inspired by human mechanisms can enhance the navigation performance of embodied agents in unseen environments. However, existing Vision-and-Language Navigation (VLN) agents lack a memory mechanism of this kind. To address this, we propose a novel architecture that equips agents with a reality-imagination hybrid memory system. This system enables agents to maintain and expand their memory through both imaginative mechanisms and navigation actions. Additionally, we design tailored pre-training tasks to develop the agent's imaginative capabilities. Our agent can imagine high-fidelity RGB images for future scenes, achieving state-of-the-art result in Success rate weighted by Path Length (SPL).

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

28 of 36 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  8. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  12. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
16 more
  1. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  2. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  3. 2020Neural Topological SLAM for Visual Navigationcited by 193
  4. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  5. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  6. 2017Learning non-maximum suppressioncited by 112
  7. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  8. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  9. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  10. 2020World-Consistent Video-to-Video Synthesiscited by 69
  11. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  12. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  13. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  14. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  15. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  16. 2020Blur, Noise, and Compression Robust Generative Adversarial Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  3. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  4. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
  5. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3
  6. 2025Enhancing Target-unspecific Tasks through a Features Matrixcited by 2
  7. 2026Robo-Cortex: A Self-Evolving Embodied Agent via Dual-Grain Cognitive Memory and Autonomous Knowledge Inductioncited by 0
  8. 2026Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Methodcited by 0