# Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning

arXiv: https://arxiv.org/abs/2310.10943 · HTML: https://mlsanity.com/paper/2310.10943/

## Key facts

- Cited by 111 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [Reinforcement Learning for UAV Attitude Control](https://mlsanity.com/paper/1804.04154/) (2018)
- [A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight](https://mlsanity.com/paper/2202.10796/) (2022)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [NeuroBEM: Hybrid Aerodynamic Quadrotor Model](https://mlsanity.com/paper/2106.08015/) (2021)
- [AlphaPilot: Autonomous Drone Racing](https://mlsanity.com/paper/2005.12813/) (2020)
- [Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight](https://mlsanity.com/paper/2307.06100/) (2023)
- [PAMPC: Perception-Aware Model Predictive Control for Quadrotors](https://mlsanity.com/paper/1804.04811/) (2018)
- [HPIPM: a high-performance quadratic programming framework for model predictive control](https://mlsanity.com/paper/2003.02547/) (2020)
- [Time-Optimal Planning for Quadrotor Waypoint Flight](https://mlsanity.com/paper/2108.04537/) (2021)
- [Model Predictive Contouring Control for Time-Optimal Quadrotor Flight](https://mlsanity.com/paper/2108.13205/) (2021)
- [Whole-Body Nonlinear Model Predictive Control Through Contacts for Quadrupeds](https://mlsanity.com/paper/1712.02889/) (2017)
- [The Artificial Intelligence behind the winning entry to the 2019 AI Robotic Racing Competition](https://mlsanity.com/paper/2109.14985/) (2021)
- [Air taxi service for urban mobility: A critical review of recent developments, future challenges, and opportunities](https://mlsanity.com/paper/2103.01768/) (2021)
- [Minimum-time trajectory generation for quadrotors in constrained environments](https://mlsanity.com/paper/1706.06478/) (2017)

## Built on by (most cited first, 30 of 111)

- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion](https://mlsanity.com/paper/2401.17583/) (2024)
- [TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction](https://mlsanity.com/paper/2405.10315/) (2024)
- [Learning Quadruped Locomotion Using Differentiable Simulation](https://mlsanity.com/paper/2403.14864/) (2024)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight](https://mlsanity.com/paper/2403.12203/) (2024)
- [Demonstrating Agile Flight from Pixels without State Estimation](https://mlsanity.com/paper/2406.12505/) (2024)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Multi-Task Reinforcement Learning for Quadrotors](https://mlsanity.com/paper/2412.12442/) (2024)
- [Learning Quadrotor Control From Visual Features Using Differentiable Simulation](https://mlsanity.com/paper/2410.15979/) (2024)
- [See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation](https://mlsanity.com/paper/2509.22653/) (2025)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight](https://mlsanity.com/paper/2501.14377/) (2025)
- [Learning Speed Adaptation for Flight in Clutter](https://mlsanity.com/paper/2403.04586/) (2024)
- [Reinforcement Learning for Collision-free Flight Exploiting Deep Collision Encoding](https://mlsanity.com/paper/2402.03947/) (2024)
- [MAVRL: Learn to Fly in Cluttered Environments with Varying Speed](https://mlsanity.com/paper/2402.08381/) (2024)
- [Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation](https://mlsanity.com/paper/2410.03076/) (2024)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [RAPTOR: A Foundation Policy for Quadrotor Control](https://mlsanity.com/paper/2509.11481/) (2025)
- [VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight](https://mlsanity.com/paper/2407.14783/) (2024)
- [Automatic Environment Shaping is the Next Frontier in RL](https://mlsanity.com/paper/2407.16186/) (2024)
- [Flying on Point Clouds with Reinforcement Learning](https://mlsanity.com/paper/2503.00496/) (2025)
- [One Net to Rule Them All: Domain Randomization in Quadcopter Racing Across Different Platforms](https://mlsanity.com/paper/2504.21586/) (2025)
- [Adaptive Stochastic Nonlinear Model Predictive Control with Look-ahead Deep Reinforcement Learning for Autonomous Vehicle Motion Control](https://mlsanity.com/paper/2311.04303/) (2023)
- [Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.16720/) (2024)
- [Environment as Policy: Learning to Race in Unseen Tracks](https://mlsanity.com/paper/2410.22308/) (2024)
- [RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation](https://mlsanity.com/paper/2502.02054/) (2025)

## Abstract (arXiv)

A central question in robotics is how to design a control system for an agile mobile robot. This paper studies this question systematically, focusing on a challenging setting: autonomous drone racing. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. We then investigated which fundamental factors have contributed to the success of RL or have limited OC. Our study indicates that the fundamental advantage of RL over OC is not that it optimizes its objective better but that it optimizes a better objective. OC decomposes the problem into planning and control with an explicit intermediate representation, such as a trajectory, that serves as an interface. This decomposition limits the range of behaviors that can be expressed by the controller, leading to inferior control performance when facing unmodeled effects. In contrast, RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses. Our findings allowed us to push an agile drone to its maximum performance, achieving a peak acceleration greater than 12 times the gravitational acceleration and a peak velocity of 108 kilometers per hour. Our policy achieved superhuman control within minutes of training on a standard workstation. This work presents a milestone in agile robotics and sheds light on the role of RL and OC in robot control.