mlsanityData to 7 Oct 2026

Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

arXiv 1909.01871 ↗First posted 4 Sep 2019cs.HC cs.AI cs.CLcited by 100

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop "Help, Anna!" (HANNA), an interactive photo-realistic simulator in which an agent fulfills object-finding tasks by requesting and interpreting natural language-and-vision assistance. An agent solving tasks in a HANNA environment can leverage simulated human assistants, called ANNA (Automatic Natural Navigation Assistants), which, upon request, provide natural language and visual instructions to direct the agent towards the goals. To address the HANNA problem, we develop a memory-augmented neural agent that hierarchically models multiple levels of decision-making, and an imitation learning algorithm that teaches the agent to avoid repeating past mistakes while simultaneously predicting its own chances of making future progress. Empirically, our approach is able to ask for help more effectively than competitive baselines and, thus, attains higher task success rate on both previously seen and previously unseen environments. We publicly release code and data at https://github.com/khanhptnk/hanna . A video demo is available at https://youtu.be/18P94aaaLKg .

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 60 references that are in the map. First posted on arXiv 4 Sep 2019.

Builds on

39 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention10 Dec
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation25 Nov

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2016Layer Normalizationcited by 5,579
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  8. 2016OpenAI Gymcited by 1,684
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
25 more
  1. 2018Learning Dexterous In-Hand Manipulationcited by 643
  2. 2018On Evaluation of Embodied Navigation Agentscited by 526
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  5. 2017Embodied Question Answeringcited by 335
  6. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  7. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  8. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  9. 2019Vision-and-Dialog Navigationcited by 246
  10. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  11. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  12. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  13. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  14. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  15. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  16. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  17. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  18. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  19. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  20. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  21. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  22. 2015Learning to Search Better Than Your Teachercited by 50
  23. 2018Talk the Walk: Navigating New York City through Grounded Dialoguecited by 44
  24. 2018Shifting the Baseline: Single Modality Performance on Visual Navigation & QAcited by 30
  25. 2019From Video Game to Real Robot: The Transfer between Action Spacescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 22; 2022: 26; 2023: 24; 2024: 17; 2025: 8; 2026: 2.

100 papers.

Sort:

By area:

  1. 2023Cognitive Architectures for Language Agentscited by 422
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  4. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  5. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  6. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  7. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  8. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  9. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  10. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  11. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  12. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
38 more
  1. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  2. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  3. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  4. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  5. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 60
  6. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  7. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  8. 2022Aerial Vision-and-Dialog Navigationcited by 48
  9. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  10. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  11. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  12. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  13. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  14. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  15. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  16. 2021Vision-Language Navigation: A Survey and Taxonomycited by 33
  17. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  18. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  19. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  20. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  21. 2021Interactive Learning from Activity Descriptioncited by 26
  22. 2022AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environmentscited by 26
  23. 2022Towards Versatile Embodied Navigationcited by 26
  24. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  25. 2021Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigationcited by 20
  26. 2022Ground then Navigate: Language-guided Navigation in Dynamic Scenescited by 20
  27. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  28. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  29. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  30. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  31. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  32. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  33. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  34. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  35. 2023Instance Tracking in 3D Scenes from Egocentric Videoscited by 17
  36. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  37. 2021Contrastive Instruction-Trajectory Learning for Vision-Language Navigationcited by 16
  38. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Models
  2. 2026IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instruction
  3. 2026ELITE: Experiential Learning and Intent-Aware Transfer for Self-improving Embodied Agents
  4. 2026Time-Aware Assistive Navigation
  5. 2021VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation
  6. 2024Embodied Instruction Following in Unknown Environments
  7. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
  8. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents