mlsanityData to 7 Oct 2026

NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

arXiv 2402.15852 ↗First posted 24 Feb 2024cs.CV cs.ROcited by 222

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-and-language navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavor to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometers, or depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision making and instruction following. We train NaVid with 510k navigation samples collected from continuous environments, including action-planning and instruction-reasoning samples, along with 763k large-scale web data. Extensive experiments show that NaVid achieves state-of-the-art performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Builds on 98 of its 118 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

98 of 118 references.

References · 98

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  12. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  13. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  14. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  15. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  16. 2018On Evaluation of Embodied Navigation Agentscited by 526
  17. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  18. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  19. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  20. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  21. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  22. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  23. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  24. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  25. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  26. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  27. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  28. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  29. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  30. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  31. 2019Vision-and-Dialog Navigationcited by 246
  32. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  33. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  34. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  35. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  36. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  37. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  38. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 82; 2026: 109.

222 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  6. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  8. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  9. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  10. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  11. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  12. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
38 more
  1. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  2. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  3. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  4. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  5. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  6. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  7. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 46
  8. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46
  9. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  10. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  11. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  12. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  13. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  14. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  15. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  16. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 34
  17. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  18. 2025Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agentscited by 30
  19. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  20. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  21. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  22. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  23. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  24. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  25. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  26. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  27. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  28. 2025FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysiscited by 21
  29. 2024Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactionscited by 18
  30. 2024Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigationcited by 18
  31. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  32. 2025Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understandingcited by 17
  33. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  34. 2026Towards Generalizable Robotic Manipulation in Dynamic Environmentscited by 17
  35. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  36. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  37. 2025TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Trackingcited by 16
  38. 2025Towards Physically Executable 3D Gaussian for Embodied Navigationcited by 16

50 most cited of 222 All 222 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks
  2. 2025FiLM-Nav: Efficient and Generalizable Navigation via VLM Fine-tuning
  3. 2025Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments
  4. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
  5. 2025A Navigation Framework Utilizing Vision-Language Models
  6. 2025NaviTrace: Evaluating Embodied Navigation of Vision-Language Models
  7. 2026NavDreamer: Video Models as Zero-Shot 3D Navigators
  8. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation