mlsanityData to 14 Oct 2026

ViNT: A Foundation Model for Visual Navigation

arXiv 2306.14846 ↗First posted 26 Jun 2023cs.RO cs.CV cs.LGcited by 222

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.

Key facts. Cited by 222 papers in the map (data through 14 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

50 of 64 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2021Variational Diffusion Modelscited by 1,153
  10. 2018On Evaluation of Embodied Navigation Agentscited by 547
  11. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  12. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  13. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  14. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  15. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  16. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  17. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  18. 2018Semi-parametric Topological Memory for Navigationcited by 209
  19. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  20. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  21. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 127
  22. 2022Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigationcited by 127
  23. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  24. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  25. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 89
  26. 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 80
  27. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 69
  28. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 68
  29. 2023SACSoN: Scalable Autonomous Control for Social Navigationcited by 61
  30. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 58
  31. 2019Deep Visual MPC-Policy Learning for Navigationcited by 56
  32. 2022TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Modelscited by 49
  33. 2019Scaling Local Control to Large-Scale Topological Navigationcited by 37
  34. 2021NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challengecited by 32
  35. 2022ExAug: Robot-Conditioned Navigation Policies via Geometric Experience Augmentationcited by 19
  36. 2023IndoorSim-to-OutdoorReal: Learning to Navigate Outdoors without any Outdoor Experiencecited by 12
  37. 2018Learning Deployable Navigation Policies at Kilometer Scale from a Single Traversalcited by 11
  38. 2022Spatio-Temporal Graph Localization Networks for Image-based Navigationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 51; 2025: 68; 2026: 92.

222 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  5. 2024Navigation World Modelscited by 272
  6. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  7. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  8. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  9. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 114
  10. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  11. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 94
  12. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
38 more
  1. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  2. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  3. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  4. 2024The Ingredients for Robotic Diffusion Transformerscited by 73
  5. 2024In-Context Imitation Learning via Next-Token Predictioncited by 70
  6. 2024Embodied Understanding of Driving Scenarioscited by 63
  7. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  8. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 60
  9. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  10. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 56
  11. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  12. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  13. 2023SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environmentscited by 53
  14. 2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
  15. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 49
  16. 2024Diffusion Transformer Policycited by 41
  17. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 41
  18. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  19. 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 38
  20. 2024Learning Manipulation by Predicting Interactioncited by 37
  21. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  22. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 34
  23. 2024Adaptive Mobile Manipulation for Articulated Objects In the Open Worldcited by 31
  24. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  25. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flightcited by 31
  26. 2024Demonstrating Agile Flight from Pixels without State Estimationcited by 31
  27. 2026Improved Baselines with Representation Autoencoderscited by 31
  28. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 29
  29. 2024NavRL: Learning Safe Flight in Dynamic Environmentscited by 26
  30. 2025NaviDiffusor: Cost-Guided Diffusion Model for Visual Navigationcited by 25
  31. 2025Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Modelscited by 25
  32. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 25
  33. 2025Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understandingcited by 19
  34. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  35. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 16
  36. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  37. 2025Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flightcited by 15
  38. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresightcited by 14

50 most cited of 222 All 222 (JSON)