mlsanityData to 7 Oct 2026

One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones

arXiv 2202.07028 ↗First posted 14 Feb 2022cs.AI cs.CL cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

We study the problem of developing autonomous agents that can follow human instructions to infer and perform a sequence of actions to complete the underlying task. Significant progress has been made in recent years, especially for tasks with short horizons. However, when it comes to long-horizon tasks with extended sequences of actions, an agent can easily ignore some instructions or get stuck in the middle of the long instructions and eventually fail the task. To address this challenge, we propose a model-agnostic milestone-based task tracker (M-TRACK) to guide the agent and monitor its progress. Specifically, we propose a milestone builder that tags the instructions with navigation and interaction milestones which the agent needs to complete step by step, and a milestone checker that systemically checks the agent's progress in its current milestone and determines when to proceed to the next. On the challenging ALFRED dataset, our M-TRACK leads to a notable 33% and 52% relative improvement in unseen success rate over two competitive base models.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 56 of its 64 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

56 of 64 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2015Pointer Networkscited by 882
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
38 more
  1. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  2. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  3. 2017Embodied Question Answeringcited by 335
  4. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  5. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  6. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  7. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  8. 2018Semi-parametric Topological Memory for Navigationcited by 203
  9. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  10. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  11. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  12. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  13. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  14. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  15. 2018Visual Semantic Navigation using Scene Priorscited by 127
  16. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  17. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  18. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  19. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  20. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  21. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  22. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  23. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  24. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  25. 2019Portuguese Named Entity Recognition using BERT-CRFcited by 73
  26. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  27. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  28. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  29. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  30. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  31. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  32. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  33. 2021Visual Navigation with Spatial Attentioncited by 43
  34. 2019A Behavioral Approach to Visual Navigation with Graph Localization Networkscited by 40
  35. 2019Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigationcited by 39
  36. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  37. 2020Factorizing Perception and Policy for Interactive Instruction Followingcited by 28
  38. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 11; 2025: 3; 2026: 1.

26 papers.

Sort:

By area:

  1. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  2. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  3. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  4. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  5. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  6. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  7. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  8. 2024Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Modelscited by 22
  9. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  10. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  11. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  12. 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Taskcited by 14
14 more
  1. 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9
  2. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  3. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  4. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  5. 2024Embodied Instruction Following in Unknown Environmentscited by 8
  6. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  7. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  8. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  9. 2024DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Controlcited by 4
  10. 2022Layout-aware Dreamer for Embodied Referring Expression Groundingcited by 3
  11. 2023RoboGPT: an intelligent agent of making embodied long-term decisions for daily instruction taskscited by 3
  12. 2024Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generationcited by 3
  13. 2024BERTologyNavigator: Advanced Question Answering with BERT-based Semanticscited by 2
  14. 2026RiverVLN: Phase-Grounded Temporal Vision--Language Navigation for Unmanned Surface Vehiclescited by 0