mlsanityData to 7 Oct 2026

Continual Vision-and-Language Navigation

arXiv 2403.15049 ↗First posted 22 Mar 2024cs.CV cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Developing Vision-and-Language Navigation (VLN) agents typically assumes a textittrain-once-deploy-once strategy, which is unrealistic as deployed agents continually encounter novel environments. To address this, we propose the Continual Vision-and-Language Navigation (CVLN) paradigm, where agents learn and adapt incrementally across multiple textitscene domains. CVLN includes two setups: Initial-instruction based CVLN for instruction-following, and Dialogue-based CVLN for dialogue-guided navigation. We also introduce two simple yet effective baselines for sequential decision-making: Perplexity Replay (PerpR), which replays difficult episodes, and Episodic Self-Replay (ESR), which stores and revisits action logits during training. Experiments show that existing continual learning methods fall short for CVLN, while PerpR and ESR achieve better performance by efficiently utilizing replay memory.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 52 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

45 of 52 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  12. 2019Gradient based sample selection for online continual learningcited by 568
33 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  5. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  6. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  7. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  8. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  9. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  10. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  13. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  14. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  15. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  16. 2018Task-Free Continual Learningcited by 219
  17. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  18. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  19. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  20. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  21. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  22. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  23. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  24. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  25. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  26. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  27. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  28. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  29. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  30. 2019The StreetLearn Environment and Datasetcited by 42
  31. 2022Iterative Vision-and-Language Navigationcited by 34
  32. 2022SYNERgy between SYNaptic consolidation and Experience Replay for general continual learningcited by 10
  33. 2022Re-examining Distillation For Continual Object Detectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Few-Shot Vision-Language Action-Incremental Policy Learningcited by 3
  2. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1