# ViNL: Visual Navigation and Locomotion Over Obstacles

arXiv: https://arxiv.org/abs/2210.14791 · HTML: https://mlsanity.com/paper/2210.14791/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots](https://mlsanity.com/paper/2111.01674/) (2021)
- [Coupling Vision and Proprioception for Navigation of Legged Robots](https://mlsanity.com/paper/2112.02094/) (2021)
- [DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learning](https://mlsanity.com/paper/1909.08399/) (2019)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Learning to Jump from Pixels](https://mlsanity.com/paper/2110.15344/) (2021)
- [Is Mapping Necessary for Realistic PointGoal Navigation?](https://mlsanity.com/paper/2206.00997/) (2022)
- [Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation](https://mlsanity.com/paper/2207.10821/) (2022)
- [Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation](https://mlsanity.com/paper/2103.08022/) (2021)
- [From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion](https://mlsanity.com/paper/2011.11722/) (2020)
- [Learning Navigation Skills for Legged Robots with Learned Robot Embeddings](https://mlsanity.com/paper/2011.12255/) (2020)

## Built on by (most cited first, 23 of 23)

- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [QUAR-VLA: Vision-Language-Action Model for Quadruped Robots](https://mlsanity.com/paper/2312.14457/) (2023)
- [GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robot](https://mlsanity.com/paper/2403.13358/) (2024)
- [Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior](https://mlsanity.com/paper/2310.01408/) (2023)
- [Spatially-Enhanced Recurrent Memory for Long-Range Mapless Navigation via End-to-End Reinforcement Learning](https://mlsanity.com/paper/2506.05997/) (2025)
- [Learning to See Physical Properties with Active Sensing Motor Policies](https://mlsanity.com/paper/2311.01405/) (2023)
- [One Filter to Deploy Them All: Robust Safety for Quadrupedal Navigation in Unknown Environments](https://mlsanity.com/paper/2412.09989/) (2024)
- [Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning](https://mlsanity.com/paper/2403.05770/) (2024)
- [DreamWaQ++: Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learning](https://mlsanity.com/paper/2409.19709/) (2024)
- [HiLMa-Res: A General Hierarchical Framework via Residual RL for Combining Quadrupedal Locomotion and Manipulation](https://mlsanity.com/paper/2407.06584/) (2024)
- [Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation](https://mlsanity.com/paper/2506.15757/) (2025)
- [GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a Gradient-Aware Mask and Semantic Constraints](https://mlsanity.com/paper/2402.14354/) (2024)
- [DeFM: Learning Foundation Representations from Depth for Robotics](https://mlsanity.com/paper/2601.18923/) (2026)
- [TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation](https://mlsanity.com/paper/2404.15256/) (2024)
- [Causality-Aware Transformer Networks for Robotic Navigation](https://mlsanity.com/paper/2409.02669/) (2024)
- [TRANS: Terrain-aware Reinforcement Learning for Agile Navigation of Quadruped Robots under Social Interactions](https://mlsanity.com/paper/2602.12724/) (2026)
- [Agile perceptive multi-skill locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2607.13579/) (2026)
- [ODTFormer: Efficient Obstacle Detection and Tracking with Stereo Cameras Based on Transformer](https://mlsanity.com/paper/2403.14626/) (2024)
- [Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning](https://mlsanity.com/paper/2503.11467/) (2025)
- [Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos](https://mlsanity.com/paper/2512.10956/) (2025)

## Abstract (arXiv)

We present Visual Navigation and Locomotion over obstacles (ViNL), which enables a quadrupedal robot to navigate unseen apartments while stepping over small obstacles that lie in its path (e.g., shoes, toys, cables), similar to how humans and pets lift their feet over objects as they walk. ViNL consists of: (1) a visual navigation policy that outputs linear and angular velocity commands that guides the robot to a goal coordinate in unfamiliar indoor environments; and (2) a visual locomotion policy that controls the robot's joints to avoid stepping on obstacles while following provided velocity commands. Both the policies are entirely "model-free", i.e. sensors-to-actions neural networks trained end-to-end. The two are trained independently in two entirely different simulators and then seamlessly co-deployed by feeding the velocity commands from the navigator to the locomotor, entirely "zero-shot" (without any co-training). While prior works have developed learning methods for visual navigation or visual locomotion, to the best of our knowledge, this is the first fully learned approach that leverages vision to accomplish both (1) intelligent navigation in new environments, and (2) intelligent visual locomotion that aims to traverse cluttered environments without disrupting obstacles. On the task of navigation to distant goals in unknown environments, ViNL using just egocentric vision significantly outperforms prior work on robust locomotion using privileged terrain maps (+32.8% success and -4.42 collisions per meter). Additionally, we ablate our locomotion policy to show that each aspect of our approach helps reduce obstacle collisions. Videos and code at http://www.joannetruong.com/projects/vinl.html