mlsanityData to 7 Oct 2026

Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation

arXiv 2607.01754 ↗First posted 2 Jul 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

On-policy exploration is a crucial component for training robust Vision-Language Navigation agents, as it exposes the policy to a broader state distribution. However, such exploration inevitably leads to trajectories that deviate from expert demonstrations, resulting in a semantic mismatch between the executed visual stream and the original language instruction. In this work, we address this challenge by introducing Phi-Nav, a unified on-policy framework that leverages hindsight reasoning to align instructions with the agent's actual exploratory journey. Specifically, Phi-Nav operates through a three-stage dual-supervision cycle: 1) the agent performs oracle-guided on-policy exploration, sampling a trajectory while learning from expert action feedback, 2) a hindsight speaker synthesizes a path-level hindsight instruction grounded in the collected visual observations, and 3) the agent conducts a second imitation pass, treating the synthesized trajectory-instruction pair as an additional expert demonstration. Through this process, Phi-Nav bridges the critical semantic supervision gap inherent in on-policy methods, transforming semantically unlabeled movement into dense training signals. Evaluations on the R2R-CE and RxR-CE benchmarks show that Phi-Nav yields competitive performance while requiring only a fraction of the expert demonstrations used by current baselines. These results underscore the necessity of semantic exploration in VLN, positioning Phi-Nav as an effective solution for training embodied agents with limited data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 64 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

54 of 64 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2025Qwen3-VL Technical Reportcited by 2,534
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
38 more
  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  2. 2019Model-Based Reinforcement Learning for Ataricited by 495
  3. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  4. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  5. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  7. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  8. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  9. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  10. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  11. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  12. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  13. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  14. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  15. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  16. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  17. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  18. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  19. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  20. 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 77
  21. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  22. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  23. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  24. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  25. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  26. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  27. 2021EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matchingcited by 36
  28. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  29. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 32
  30. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  31. 2018Hierarchical Reinforcement Learning with Hindsightcited by 26
  32. 2018Energy-Based Hindsight Experience Prioritizationcited by 20
  33. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  34. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  35. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  36. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  37. 2019HIGhER : Improving instruction following with Hindsight Generation for Experience Replaycited by 13
  38. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.