mlsanityData to 14 Oct 2026

Vision-Language Navigation with Energy-Based Policy

arXiv 2410.14250 ↗First posted 18 Oct 2024cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Vision-language navigation (VLN) requires an agent to execute actions following human instructions. Existing VLN models are optimized through expert demonstrations by supervised behavioural cloning or incorporating manual reward engineering. While straightforward, these efforts overlook the accumulation of errors in the Markov decision process, and struggle to match the distribution of the expert policy. Going beyond this, we propose an Energy-based Navigation Policy (ENP) to model the joint state-action distribution using an energy-based model. At each step, low energy values correspond to the state-action pairs that the expert is most likely to perform, and vice versa. Theoretically, the optimization objective is equivalent to minimizing the forward divergence between the occupancy measure of the expert and ours. Consequently, ENP learns to globally align with the expert policy by maximizing the likelihood of the actions and modeling the dynamics of the navigation states in a collaborative manner. With a variety of VLN architectures, ENP achieves promising performances on R2R, REVERIE, RxR, and R2R-CE, unleashing the power of existing VLN models.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 85 of its 110 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

85 of 110 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2016Generative Adversarial Imitation Learningcited by 1,597
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  11. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
38 more
  1. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  2. 2016End to End Learning for Self-Driving Carscited by 1,009
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2019Model-Based Reinforcement Learning for Ataricited by 502
  6. 2021Implicit Behavioral Cloningcited by 452
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  8. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  10. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  13. 2020Learning to Explore using Active Neural SLAMcited by 316
  14. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  15. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  16. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  17. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  18. 2021How to Train Your Energy-Based Modelscited by 274
  19. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  20. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  21. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  22. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  23. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  24. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  25. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  26. 2019Efficient Exploration via State Marginal Matchingcited by 188
  27. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  28. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  29. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  30. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  31. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  32. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  33. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  34. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  35. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  36. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  37. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  38. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  2. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  3. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 19
  4. 2025MonoDream: Monocular Vision-Language Navigation with Panoramic Dreamingcited by 16
  5. 2025View Invariant Learning for Vision-Language Navigation in Continuous Environmentscited by 6
  6. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  7. 2025History-Enhanced Two-Stage Transformer for Aerial Vision-and-Language Navigationcited by 6
  8. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  9. 2026Let's Reward Step-by-Step: Step-Aware Contrastive Alignment for Vision-Language Navigation in Continuous Environmentscited by 3
  10. 2026Does Peer Observation Help? Vision-Sharing Collaboration for Vision-Language Navigationcited by 2
  11. 2026Goal2Pixel: Grounding Goals to Pixels for Vision-Language Navigationcited by 2
  12. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
7 more
  1. 2025Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarioscited by 1
  2. 2025Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understandingcited by 1
  3. 2026Dynamic Topology Awareness: Breaking the Granularity Rigidity in Vision-Language Navigationcited by 1
  4. 2026Uncertainty-Aware Gaussian Map for Vision-Language Navigationcited by 1
  5. 2026PM-Nav: Priori-Map Guided Embodied Navigation in Functional Buildingscited by 0
  6. 2026Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigationcited by 0
  7. 2026Seek Before You Move: Evidence Seeking for Progress Grounding in Vision-Language Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
  2. 2026ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning
  3. 2026LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory
  4. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation
  5. 2025SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization
  6. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigation
  7. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation
  8. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks