mlsanityData to 14 Oct 2026

Vision and Language Navigation in the Real World via Online Visual Language Mapping

arXiv 2310.10822 ↗First posted 16 Oct 2023cs.RO cs.CV cs.SYcited by 11

Not on the map. Why?

Abstract (arXiv)

Navigating in unseen environments is crucial for mobile robots. Enhancing them with the ability to follow instructions in natural language will further improve navigation efficiency in unseen cases. However, state-of-the-art (SOTA) vision-and-language navigation (VLN) methods are mainly evaluated in simulation, neglecting the complex and noisy real world. Directly transferring SOTA navigation policies trained in simulation to the real world is challenging due to the visual domain gap and the absence of prior knowledge about unseen environments. In this work, we propose a novel navigation framework to address the VLN task in the real world. Utilizing the powerful foundation models, the proposed framework includes four key components: (1) an LLMs-based instruction parser that converts the language instruction into a sequence of pre-defined macro-action descriptions, (2) an online visual-language mapper that builds a real-time visual-language map to maintain a spatial and semantic understanding of the unseen environment, (3) a language indexing-based localizer that grounds each macro-action description into a waypoint location on the map, and (4) a DD-PPO-based local controller that predicts the action. We evaluate the proposed pipeline on an Interbotix LoCoBot WX250 in an unseen lab environment. Without any fine-tuning, our pipeline significantly outperforms the SOTA VLN baseline in the real world.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

30 of 33 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  6. 2022Language-driven Semantic Segmentationcited by 717
  7. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  9. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  10. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  11. 2022Visual Language Maps for Robot Navigationcited by 365
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
18 more
  1. 2020Learning to Explore using Active Neural SLAMcited by 316
  2. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  4. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  5. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  6. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  7. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 129
  8. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  9. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 89
  10. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  11. 2021The Road to Know-Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigationcited by 61
  12. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  13. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  14. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  15. 2020Semantic Curiosity for Active Visual Learningcited by 45
  16. 2022Is Mapping Necessary for Realistic PointGoal Navigation?cited by 35
  17. 2021Differentiable SLAM-net: Learning Particle SLAM for Visual Navigationcited by 34
  18. 2021Image-Goal Navigation in Complex Environments via Modular Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 1; 2026: 3.

11 papers.

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  3. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  4. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 19
  5. 2024Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environmentscited by 10
  6. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  7. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  8. 2024NOLO: Navigate Only Look Oncecited by 3
  9. 2026Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigationcited by 1
  10. 2026MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planningcited by 0
  11. 2026Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Modelscited by 0