mlsanityData to 14 Oct 2026

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

arXiv 2407.07775 ↗First posted 10 Jul 2024cs.RO cs.AIcited by 73

Not on the map. Why?

Cited by 20 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836m^2 real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should I return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: https://youtu.be/-Tof__Q8_5s

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

46 of 63 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  8. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  9. 2018On Evaluation of Embodied Navigation Agentscited by 547
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  11. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
34 more
  1. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  2. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  3. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  4. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  5. 2019Vision-and-Dialog Navigationcited by 250
  6. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  7. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  8. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  9. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  10. 2018Semi-parametric Topological Memory for Navigationcited by 209
  11. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  12. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  13. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  14. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  15. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  16. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  17. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
  18. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  19. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  20. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 107
  21. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  22. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  23. 2023Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skillcited by 77
  24. 2022Offline Visual Representation Learning for Embodied Navigationcited by 75
  25. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 66
  26. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  27. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  28. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  29. 2023NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshescited by 53
  30. 2021Grouptron: Dynamic Multi-Scale Graph Convolutional Networks for Group-Aware Dense Crowd Trajectory Forecastingcited by 19
  31. 2024MATRIX: Multi-Agent Trajectory Generation with Diverse Contextscited by 9
  32. 2018Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Controlcited by 7
  33. 2020Cascade Attribute Network: Decomposing Reinforcement Learning Control Policies using Hierarchical Neural Networkscited by 4
  34. 2023Distributed Multi-agent Interaction Generation with Imagined Potential Gamescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 33; 2026: 25.

73 papers.

Sort:

By area:

  1. 2025Agentic Large Language Models, a surveycited by 108
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  3. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  4. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
  5. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  6. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  7. 2025LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instructioncited by 43
  8. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 39
  9. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  10. 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 38
  11. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  12. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 36
38 more
  1. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 34
  2. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 34
  3. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 32
  4. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 31
  5. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 29
  6. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  7. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 21
  8. 2025Describe Anything Anywhere At Any Momentcited by 19
  9. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 18
  10. 2026Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoningcited by 18
  11. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 16
  12. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  13. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 14
  14. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 12
  15. 2025Air-Ground Collaboration for Language-Specified Missions in Unknown Environmentscited by 11
  16. 2026TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environmentscited by 10
  17. 2024UniRS: Unifying Multi-temporal Remote Sensing Tasks through Vision Language Modelscited by 9
  18. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 9
  19. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 8
  20. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  21. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 7
  22. 2025TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goalscited by 7
  23. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulationcited by 6
  24. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 6
  25. 2024NL-Eye: Abductive NLI for Imagescited by 5
  26. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuningcited by 5
  27. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  28. 2025Astra: Toward General-Purpose Mobile Robots via Hierarchical Multimodal Learningcited by 4
  29. 2025Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environmentscited by 4
  30. 2025ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusioncited by 4
  31. 2025Human-like Navigation in a World Built for Humanscited by 4
  32. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  33. 2026MapDream: Task-Driven Map Learning for Vision-Language Navigationcited by 4
  34. 2026AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Modelscited by 4
  35. 2024SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Modelscited by 3
  36. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answeringcited by 3
  37. 2025MG-Nav: Dual-Scale Visual Navigation via Sparse Spatial Memorycited by 3
  38. 2025Scaling Spatial Reasoning in MLLMs through Programmatic Data Synthesiscited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  2. 2026What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
  3. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
  4. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
  5. 2025GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoning
  6. 2024InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models
  7. 2024Navigation with VLM framework: Towards Going to Any Language
  8. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents