# Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing

arXiv: https://arxiv.org/abs/1810.06224 · HTML: https://mlsanity.com/paper/1810.06224/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2018.

## Built on by (most cited first, 21 of 21)

- [The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI](https://mlsanity.com/paper/2103.14025/) (2021)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Policy Search for Model Predictive Control with Application to Agile Drone Flight](https://mlsanity.com/paper/2112.03850/) (2021)
- [SelfD: Self-Learning Large-Scale Driving Policies From the Web](https://mlsanity.com/paper/2204.10320/) (2022)
- [Finding Fallen Objects Via Asynchronous Audio-Visual Integration](https://mlsanity.com/paper/2207.03483/) (2022)
- [Reinforcement Learning for Collision-free Flight Exploiting Deep Collision Encoding](https://mlsanity.com/paper/2402.03947/) (2024)
- [MOPA: Modular Object Navigation with PointGoal Agents](https://mlsanity.com/paper/2304.03696/) (2023)
- [Vision-based Learning for Drones: A Survey](https://mlsanity.com/paper/2312.05019/) (2023)
- [Learning Multi-Pursuit Evasion for Safe Targeted Navigation of Drones](https://mlsanity.com/paper/2304.03443/) (2023)
- [The Artificial Intelligence behind the winning entry to the 2019 AI Robotic Racing Competition](https://mlsanity.com/paper/2109.14985/) (2021)
- [Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents](https://mlsanity.com/paper/2201.02569/) (2022)
- [Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts](https://mlsanity.com/paper/2311.01457/) (2023)
- [One-shot Visual Imitation via Attributed Waypoints and Demonstration Augmentation](https://mlsanity.com/paper/2302.04856/) (2023)
- [CUAHN-VIO: Content-and-Uncertainty-Aware Homography Network for Visual-Inertial Odometry](https://mlsanity.com/paper/2208.13935/) (2022)
- [Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots](https://mlsanity.com/paper/2106.13280/) (2021)
- [Design and implementation of a parsimonious neuromorphic PID for onboard altitude control for MAVs using neuromorphic processors](https://mlsanity.com/paper/2109.10199/) (2021)
- [A Map-free Deep Learning-based Framework for Gate-to-Gate Monocular Visual Navigation aboard Miniaturized Aerial Vehicles](https://mlsanity.com/paper/2503.05251/) (2025)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Why Change Your Controller When You Can Change Your Planner: Drag-Aware Trajectory Generation for Quadrotor Systems](https://mlsanity.com/paper/2401.04960/) (2024)
- [Continual Learning for Robust Gate Detection under Dynamic Lighting in Autonomous Drone Racing](https://mlsanity.com/paper/2405.01054/) (2024)

## Abstract (arXiv)

Autonomous micro aerial vehicles still struggle with fast and agile maneuvers, dynamic environments, imperfect sensing, and state estimation drift. Autonomous drone racing brings these challenges to the fore. Human pilots can fly a previously unseen track after a handful of practice runs. In contrast, state-of-the-art autonomous navigation algorithms require either a precise metric map of the environment or a large amount of training data collected in the track of interest. To bridge this gap, we propose an approach that can fly a new track in a previously unseen environment without a precise map or expensive data collection. Our approach represents the global track layout with coarse gate locations, which can be easily estimated from a single demonstration flight. At test time, a convolutional network predicts the poses of the closest gates along with their uncertainty. These predictions are incorporated by an extended Kalman filter to maintain optimal maximum-a-posteriori estimates of gate locations. This allows the framework to cope with misleading high-variance estimates that could stem from poor observability or lack of visible gates. Given the estimated gate poses, we use model predictive control to quickly and accurately navigate through the track. We conduct extensive experiments in the physical world, demonstrating agile and robust flight through complex and diverse previously-unseen race tracks. The presented approach was used to win the IROS 2018 Autonomous Drone Race Competition, outracing the second-placing team by a factor of two.