mlsanityData to 14 Oct 2026

NaVILA: Legged Robot Vision-Language-Action Model for Navigation

arXiv 2412.04453 ↗First posted 5 Dec 2024cs.RO cs.CVcited by 211

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes. However, it is non-trivial to translate human language instructions all the way to low-level leg joint actions. We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills. Instead of directly predicting low-level actions from VLA, NaVILA first generates mid-level actions with spatial information in the form of language, (e.g., "moving forward 75cm"), which serves as an input for a visual locomotion RL policy for execution. NaVILA substantially improves previous approaches on existing benchmarks. The same advantages are demonstrated in our newly developed benchmarks with IsaacLab, featuring more realistic scenes, low-level controls, and real-world robot experiments. We show more results at https://navila-bot.github.io/

Key facts. Cited by 211 papers in the map (data through 14 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 96 of its 110 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

96 of 110 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  2. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robots22 Dec
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 93

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
38 more
  1. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  4. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  5. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  6. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  7. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  8. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  9. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  10. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  11. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  13. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  14. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  15. 2020Learning to Explore using Active Neural SLAMcited by 316
  16. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  17. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  18. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  19. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  20. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  21. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  22. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  23. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  24. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  25. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  26. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  27. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  28. 2020Neural Topological SLAM for Visual Navigationcited by 195
  29. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  30. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  31. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  32. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  33. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  34. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169
  35. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  36. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  37. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  38. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 86; 2026: 124.

211 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  3. 2025Real-Time Execution of Action Chunking Flow Policiescited by 216
  4. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 118
  5. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 106
  6. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  7. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  8. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  9. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  10. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  11. 2025Training-Time Action Conditioning for Efficient Real-Time Chunkingcited by 64
  12. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
38 more
  1. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  2. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 49
  3. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  4. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 47
  5. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 46
  6. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  7. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 44
  8. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 43
  9. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  10. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  11. 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 38
  12. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  13. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  14. 20253D Aware Region Prompted Vision Language Modelcited by 33
  15. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  16. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 29
  17. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 27
  18. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 26
  19. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 25
  20. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 25
  21. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 23
  22. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  23. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 23
  24. 2025VLA-AN: An Efficient and Onboard Vision-Language-Action Framework for Aerial Navigation in Complex Environmentscited by 23
  25. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoningcited by 22
  26. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 17
  27. 2025TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Trackingcited by 17
  28. 2025Towards Physically Executable 3D Gaussian for Embodied Navigationcited by 17
  29. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  30. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  31. 2025MonoDream: Monocular Vision-Language Navigation with Panoramic Dreamingcited by 16
  32. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 16
  33. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 15
  34. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  35. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 14
  36. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 13
  37. 2025NaviTrace: Evaluating Embodied Navigation of Vision-Language Modelscited by 13
  38. 2025Dexbotic: Open-Source Vision-Language-Action Toolboxcited by 12

50 most cited of 211 All 211 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LaViRA: Language-Vision-Robot Actions Translation for Zero-Shot Vision Language Navigation in Continuous Environments
  2. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
  3. 2026KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition
  4. 2026LocoVLM: Grounding Vision and Language for Adapting Versatile Legged Locomotion Policies
  5. 2025Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models
  6. 2026GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
  7. 2026SimVLA: A Simple VLA Baseline for Robotic Manipulation
  8. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Survey