# Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

arXiv: https://arxiv.org/abs/2201.02569 · HTML: https://mlsanity.com/paper/2201.02569/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 7 Jan 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Deep Drone Acrobatics](https://mlsanity.com/paper/2006.05768/) (2020)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [An Autonomous Drone for Search and Rescue in Forests using Airborne Optical Sectioning](https://mlsanity.com/paper/2105.04328/) (2021)
- [AlphaPilot: Autonomous Drone Racing](https://mlsanity.com/paper/2005.12813/) (2020)
- [RAPTOR: Robust and Perception-aware Trajectory Replanning for Quadrotor Fast Flight](https://mlsanity.com/paper/2007.03465/) (2020)
- [Time-Optimal Planning for Quadrotor Waypoint Flight](https://mlsanity.com/paper/2108.04537/) (2021)
- [FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation](https://mlsanity.com/paper/1905.11377/) (2019)
- [Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing](https://mlsanity.com/paper/1810.06224/) (2018)
- [Fast, Autonomous Flight in GPS-Denied and Cluttered Environments](https://mlsanity.com/paper/1712.02052/) (2017)
- [AirSim Drone Racing Lab](https://mlsanity.com/paper/2003.05654/) (2020)
- [OIL: Observational Imitation Learning](https://mlsanity.com/paper/1803.01129/) (2018)
- [Model Predictive Control for Micro Aerial Vehicles: A Survey](https://mlsanity.com/paper/2011.11104/) (2020)
- [Human-Piloted Drone Racing: Visual Processing and Control](https://mlsanity.com/paper/2103.04672/) (2021)
- [Autonomous drone race: A computationally efficient vision-based navigation and control strategy](https://mlsanity.com/paper/1809.05958/) (2018)
- [Human Visual Attention Prediction Boosts Learning & Performance of Autonomous Driving Agents](https://mlsanity.com/paper/1909.05003/) (2019)
- [DeepRacing: Parameterized Trajectories for Autonomous Racing](https://mlsanity.com/paper/2005.05178/) (2020)
- [Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-based UAV Racing](https://mlsanity.com/paper/1904.08801/) (2019)
- [GPA-Teleoperation: Gaze Enhanced Perception-aware Safe Assistive Aerial Teleoperation](https://mlsanity.com/paper/2109.04907/) (2021)
- [Expertise Affects Drone Racing Performance](https://mlsanity.com/paper/2109.07307/) (2021)

## Built on by (most cited first, 7 of 7)

- [What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? A Comprehensive Study](https://mlsanity.com/paper/2412.11764/) (2024)
- [The Power of Input: Benchmarking Zero-Shot Sim-To-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control](https://mlsanity.com/paper/2410.07686/) (2024)
- [Continual Learning for Robust Gate Detection under Dynamic Lighting in Autonomous Drone Racing](https://mlsanity.com/paper/2405.01054/) (2024)
- [HDVIO2.0: Wind and Disturbance Estimation with Hybrid Dynamics VIO](https://mlsanity.com/paper/2504.00969/) (2025)
- [State-Conditional Adversarial Learning: An Off-Policy Visual Domain Transfer Method for End-to-End Imitation Learning](https://mlsanity.com/paper/2512.05335/) (2025)
- [Neurosymbolic Imitation Learning with Human Guidance: A Privileged Information Approach](https://mlsanity.com/paper/2605.07166/) (2026)
- [Fast Human Attention Prediction for Fixation-guided Active Perception in Autonomous Navigation](https://mlsanity.com/paper/2606.20491/) (2026)

## Abstract (arXiv)

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural network performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Comparing success rates for completing a challenging race track by autonomous flight, our results show that the attention-prediction based controller (88% success rate) outperforms the RGB-image (61% success rate) and feature-tracks (55% success rate) controller baselines. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.