mlsanityData to 7 Oct 2026

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

arXiv 1711.07280 ↗First posted 20 Nov 2017cs.CV cs.AI cs.CLcited by 1,013

Not on the map. Why?

Cited by 79 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matterport3D Simulator -- a large-scale reinforcement learning environment based on real imagery. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -- the Room-to-Room (R2R) dataset.

Key facts. Cited by 1,013 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 2; 2020: 2; 2021: 123; 2022: 150; 2023: 157; 2024: 160; 2025: 184; 2026: 232.

1,013 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  6. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  7. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  8. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  9. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  10. 2023Learning Interactive Real-World Simulatorscited by 440
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  12. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
38 more
  1. 2022Visual Language Maps for Robot Navigationcited by 362
  2. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  3. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  4. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  5. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  6. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  7. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  8. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  9. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  10. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  11. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  12. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  13. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  14. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  15. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  16. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  17. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  18. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  19. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  20. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  21. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  22. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  23. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  24. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  25. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  26. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  27. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  28. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  29. 2023Large Language Models for Robotics: A Surveycited by 132
  30. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  31. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  32. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  33. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  34. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  35. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  36. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  37. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  38. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117

50 most cited of 1,013 All 1,013 (JSON)