# VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation

arXiv: https://arxiv.org/abs/2105.09371 · HTML: https://mlsanity.com/paper/2105.09371/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 19 May 2021.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 20 of 20)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [Semantic Visual Navigation by Watching YouTube Videos](https://mlsanity.com/paper/2006.10034/) (2020)
- [Model-Based Inverse Reinforcement Learning from Visual Demonstrations](https://mlsanity.com/paper/2010.09034/) (2020)
- [Imitation Learning from Video by Leveraging Proprioception](https://mlsanity.com/paper/1905.09335/) (2019)
- [RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration](https://mlsanity.com/paper/1906.07372/) (2019)
- [Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation](https://mlsanity.com/paper/1909.12971/) (2019)
- [Navigation by Imitation in a Pedestrian-Rich Environment](https://mlsanity.com/paper/1811.00506/) (2018)

## Built on by (most cited first, 30 of 31)

- [Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation](https://mlsanity.com/paper/2203.15041/) (2022)
- [Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations](https://mlsanity.com/paper/2303.01497/) (2023)
- [Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation](https://mlsanity.com/paper/2112.05251/) (2021)
- [Towards Generalist Robot Learning from Internet Video: A Survey](https://mlsanity.com/paper/2404.19664/) (2024)
- [K-VIL: Keypoints-based Visual Imitation Learning](https://mlsanity.com/paper/2209.03277/) (2022)
- [VI-IKD: High-Speed Accurate Off-Road Navigation using Learned Visual-Inertial Inverse Kinodynamics](https://mlsanity.com/paper/2203.15983/) (2022)
- [CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding](https://mlsanity.com/paper/2310.06214/) (2023)
- [High-Speed Accurate Robot Control using Learned Forward Kinodynamics and Non-linear Least Squares Optimization](https://mlsanity.com/paper/2206.08487/) (2022)
- [STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience](https://mlsanity.com/paper/2309.15302/) (2023)
- [Domain Adaptive Imitation Learning with Visual Observation](https://mlsanity.com/paper/2312.00548/) (2023)
- [Spatially-Enhanced Recurrent Memory for Long-Range Mapless Navigation via End-to-End Reinforcement Learning](https://mlsanity.com/paper/2506.05997/) (2025)
- [Adversarial Imitation Learning from Video using a State Observer](https://mlsanity.com/paper/2202.00243/) (2022)
- [Wait, That Feels Familiar: Learning to Extrapolate Human Preferences for Preference Aligned Path Planning](https://mlsanity.com/paper/2309.09912/) (2023)
- [Reinforcement Learning for Wheeled Mobility on Vertically Challenging Terrain](https://mlsanity.com/paper/2409.02383/) (2024)
- [Visual Hindsight Self-Imitation Learning for Interactive Navigation](https://mlsanity.com/paper/2312.03446/) (2023)
- [Learning with Chemical versus Electrical Synapses -- Does it Make a Difference?](https://mlsanity.com/paper/2401.08602/) (2023)
- [Hierarchical end-to-end autonomous navigation through few-shot waypoint detection](https://mlsanity.com/paper/2409.14633/) (2024)
- [CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction](https://mlsanity.com/paper/2410.01273/) (2024)
- [One-Shot Learning of Visual Path Navigation for Autonomous Vehicles](https://mlsanity.com/paper/2306.08865/) (2023)
- [On Combining Expert Demonstrations in Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2307.10810/) (2023)
- [Active Third-Person Imitation Learning](https://mlsanity.com/paper/2312.16365/) (2023)
- [Dyna-LfLH: Learning Agile Navigation in Dynamic Environments from Learned Hallucination](https://mlsanity.com/paper/2403.17231/) (2024)
- [Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaboration](https://mlsanity.com/paper/2511.08732/) (2025)
- [Lift, Splat, Map: Lifting Foundation Masks for Label-Free Semantic Scene Completion](https://mlsanity.com/paper/2407.03425/) (2024)
- [Collision Avoidance for Multiple UAVs in Unknown Scenarios with Causal Representation Disentanglement](https://mlsanity.com/paper/2407.04064/) (2024)
- [Aligning Robot Navigation Behaviors with Human Intentions and Preferences](https://mlsanity.com/paper/2409.18982/) (2024)
- [Reliable Semantic Understanding for Real World Zero-shot Object Goal Navigation](https://mlsanity.com/paper/2410.21926/) (2024)
- [Feasibility-aware Imitation Learning from Observation with Multimodal Feedback](https://mlsanity.com/paper/2602.15351/) (2026)
- [SR-Nav: Spatial Relationships Matter for Zero-shot Object Goal Navigation](https://mlsanity.com/paper/2603.18443/) (2026)
- [HumAIN: Human-Aware Implicit Social Robot Navigation](https://mlsanity.com/paper/2607.07357/) (2026)

## Abstract (arXiv)

While imitation learning for vision based autonomous mobile robot navigation has recently received a great deal of attention in the research community, existing approaches typically require state action demonstrations that were gathered using the deployment platform. However, what if one cannot easily outfit their platform to record these demonstration signals or worse yet the demonstrator does not have access to the platform at all? Is imitation learning for vision based autonomous navigation even possible in such scenarios? In this work, we hypothesize that the answer is yes and that recent ideas from the Imitation from Observation (IfO) literature can be brought to bear such that a robot can learn to navigate using only ego centric video collected by a demonstrator, even in the presence of viewpoint mismatch. To this end, we introduce a new algorithm, Visual Observation only Imitation Learning for Autonomous navigation (VOILA), that can successfully learn navigation policies from a single video demonstration collected from a physically different agent. We evaluate VOILA in the photorealistic AirSim simulator and show that VOILA not only successfully imitates the expert, but that it also learns navigation policies that can generalize to novel environments. Further, we demonstrate the effectiveness of VOILA in a real world setting by showing that it allows a wheeled Jackal robot to successfully imitate a human walking in an environment using a video recorded using a mobile phone camera.