# Learn by Observation: Imitation Learning for Drone Patrolling from Videos of A Human Navigator

arXiv: https://arxiv.org/abs/2008.13193 · HTML: https://mlsanity.com/paper/2008.13193/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 30 Aug 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 16 of 16)

- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017)
- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Discriminative Correlation Filter with Channel and Spatial Reliability](https://mlsanity.com/paper/1611.08461/) (2016)
- [Variational End-to-End Navigation and Localization](https://mlsanity.com/paper/1811.10119/) (2018)
- [Uncertainty-Aware Learning from Demonstration using Mixture Density Networks with Sampling-Free Variance Modeling](https://mlsanity.com/paper/1709.02249/) (2017)
- [Toward Low-Flying Autonomous MAV Trail Navigation using Deep Neural Networks for Environmental Awareness](https://mlsanity.com/paper/1705.02550/) (2017)
- [Efficient Optical flow and Stereo Vision for Velocity Estimation and Obstacle Avoidance on an Autonomous Pocket Drone](https://mlsanity.com/paper/1612.06702/) (2016)
- [Deep Neural Network for Real-Time Autonomous Indoor Navigation](https://mlsanity.com/paper/1511.04668/) (2015)
- [Learning Actions from Human Demonstration Video for Robotic Manipulation](https://mlsanity.com/paper/1909.04312/) (2019)
- [Learning Transferable UAV for Forest Visual Perception](https://mlsanity.com/paper/1806.03626/) (2018)

## Built on by (most cited first, 5 of 5)

- [Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations](https://mlsanity.com/paper/2303.01497/) (2023)
- [Aerial Vision-and-Dialog Navigation](https://mlsanity.com/paper/2205.12219/) (2022)
- [AeroDuo: Aerial Duo for UAV-based Vision and Language Navigation](https://mlsanity.com/paper/2508.15232/) (2025)
- [Learning with Chemical versus Electrical Synapses -- Does it Make a Difference?](https://mlsanity.com/paper/2401.08602/) (2023)
- [Multi-model fusion for Aerial Vision and Dialog Navigation based on human attention aids](https://mlsanity.com/paper/2308.14064/) (2023)

## Abstract (arXiv)

We present an imitation learning method for autonomous drone patrolling based only on raw videos. Different from previous methods, we propose to let the drone learn patrolling in the air by observing and imitating how a human navigator does it on the ground. The observation process enables the automatic collection and annotation of data using inter-frame geometric consistency, resulting in less manual effort and high accuracy. Then a newly designed neural network is trained based on the annotated data to predict appropriate directions and translations for the drone to patrol in a lane-keeping manner as humans. Our method allows the drone to fly at a high altitude with a broad view and low risk. It can also detect all accessible directions at crossroads and further carry out the integration of available user instructions and autonomous patrolling control commands. Extensive experiments are conducted to demonstrate the accuracy of the proposed imitating learning process as well as the reliability of the holistic system for autonomous drone navigation. The codes, datasets as well as video demonstrations are available at https://vsislab.github.io/uavpatrol