mlsanityData to 7 Oct 2026

Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention

arXiv 1812.04155 ↗First posted 10 Dec 2018cs.LG cs.CL cs.CVcited by 77

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level end-goals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines in both seen and unseen environments. Our code and data are publicly available at https://github.com/debadeepta/vnla .

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 67 references that are in the map. First posted on arXiv 10 Dec 2018.

Builds on

41 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation25 Nov
  2. 2018Guiding Policies with Language via Meta-Learning19 Nov

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
27 more
  1. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  2. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  3. 2017Embodied Question Answeringcited by 335
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  5. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  6. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  7. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  8. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  9. 2016Modeling Coverage for Neural Machine Translationcited by 110
  10. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  11. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  12. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  13. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  14. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  15. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  16. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  17. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  18. 2015Learning to Search Better Than Your Teachercited by 50
  19. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44
  20. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  21. 2017Data-driven Planning via Imitation Learningcited by 36
  22. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  23. 2017Representation Learning for Grounded Spatial Reasoningcited by 23
  24. 2017HoME: a Household Multimodal Environmentcited by 20
  25. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  26. 2014RoboBrain: Large-Scale Knowledge Engine for Robotscited by 13
  27. 2018Safe Navigation with Human Instructions in Complex Scenescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 16; 2022: 14; 2023: 18; 2024: 11; 2025: 11; 2026: 5.

77 papers.

Sort:

By area:

  1. 2023Cognitive Architectures for Language Agentscited by 422
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  3. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  4. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  5. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  6. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  7. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  8. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  9. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  10. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  11. 2021Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learningcited by 53
  12. 2022Aerial Vision-and-Dialog Navigationcited by 48
38 more
  1. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  2. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  3. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  4. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  5. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  6. 2021Interactive Learning from Activity Descriptioncited by 26
  7. 2022AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environmentscited by 26
  8. 2022Towards Versatile Embodied Navigationcited by 26
  9. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  10. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 25
  11. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  12. 2021Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigationcited by 20
  13. 2022Ground then Navigate: Language-guided Navigation in Dynamic Scenescited by 20
  14. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  15. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  16. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  17. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18
  18. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  19. 2021Contrastive Instruction-Trajectory Learning for Vision-Language Navigationcited by 16
  20. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  21. 2021Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigationcited by 15
  22. 2025MonoDream: Monocular Vision-Language Navigation with Panoramic Dreamingcited by 15
  23. 2021CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Imagescited by 14
  24. 2023CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environmentscited by 14
  25. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  26. 2023CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Groundingcited by 13
  27. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  28. 2025UAV-VLN: End-to-End Vision Language guided Navigation for UAVscited by 11
  29. 2022Reasoning about Actions over Visual and Linguistic Modalities: A Surveycited by 10
  30. 2023CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigationcited by 10
  31. 2023HabiCrowd: A High Performance Simulator for Crowd-Aware Visual Navigationcited by 9
  32. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  33. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
  34. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  35. 2025CogDDN: A Cognitive Demand-Driven Navigation with Decision Optimization and Dual-Process Thinkingcited by 7
  36. 2023Chain-of-Symbol Prompting Elicits Planning in Large Langauge Modelscited by 6
  37. 2025HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactionscited by 6
  38. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Accessible Instruction-Following Agent
  2. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
  3. 2020Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
  4. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning
  5. 2025A Navigation Framework Utilizing Vision-Language Models
  6. 2026AgentVLN: Towards Agentic Vision-and-Language Navigation
  7. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
  8. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning