# What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? A Comprehensive Study

arXiv: https://arxiv.org/abs/2412.11764 · HTML: https://mlsanity.com/paper/2412.11764/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flight](https://mlsanity.com/paper/2202.10796/) (2022)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [NeuroBEM: Hybrid Aerodynamic Quadrotor Model](https://mlsanity.com/paper/2106.08015/) (2021)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories](https://mlsanity.com/paper/1712.02402/) (2017)
- [Learning to Fly in Seconds](https://mlsanity.com/paper/2311.13081/) (2023)
- [PAMPC: Perception-Aware Model Predictive Control for Quadrotors](https://mlsanity.com/paper/1804.04811/) (2018)
- [Learning a Single Near-hover Position Controller for Vastly Different Quadcopters](https://mlsanity.com/paper/2209.09232/) (2022)
- [Performance, Precision, and Payloads: Adaptive Nonlinear MPC for Quadrotors](https://mlsanity.com/paper/2109.04210/) (2021)
- [Learning Minimum-Time Flight in Cluttered Environments](https://mlsanity.com/paper/2203.15052/) (2022)
- [DATT: Deep Adaptive Trajectory Tracking for Quadrotor Control](https://mlsanity.com/paper/2310.09053/) (2023)
- [Aggressive Quadrotor Flight Using Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2203.14033/) (2022)
- [Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents](https://mlsanity.com/paper/2201.02569/) (2022)
- [The Power of Input: Benchmarking Zero-Shot Sim-To-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control](https://mlsanity.com/paper/2410.07686/) (2024)
- [Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors](https://mlsanity.com/paper/2201.01369/) (2022)
- [Neural Internal Model Control: Learning a Robust Control Policy via Predictive Error Feedback](https://mlsanity.com/paper/2411.13079/) (2024)

## Built on by (most cited first, 11 of 11)

- [Leveling the Playing Field: Carefully Comparing Classical and Learned Controllers for Quadrotor Trajectory Tracking](https://mlsanity.com/paper/2506.17832/) (2025)
- [VolleyBots: A Testbed for Multi-Drone Volleyball Game Combining Motion Control and Strategic Play](https://mlsanity.com/paper/2502.01932/) (2025)
- [DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning](https://mlsanity.com/paper/2509.10247/) (2025)
- [Hysteresis-Aware Neural Network Modeling and Whole-Body Reinforcement Learning Control of Soft Robots](https://mlsanity.com/paper/2504.13582/) (2025)
- [Towards Task-Oriented Flying: Framework, Infrastructure, and Principles](https://mlsanity.com/paper/2504.15129/) (2025)
- [JuggleRL: Mastering Ball Juggling with a Quadrotor via Deep Reinforcement Learning](https://mlsanity.com/paper/2509.24892/) (2025)
- [Dynamics-Invariant Quadrotor Control using Scale-Aware Deep Reinforcement Learning](https://mlsanity.com/paper/2503.09622/) (2025)
- [PerchRL: Vision-Based Agile Perching on Inclined Platforms under Rapid and Irregular Motion](https://mlsanity.com/paper/2606.03441/) (2026)
- [Lightweight Safe Reinforcement Learning for End-to-End UAV Navigation](https://mlsanity.com/paper/2607.01794/) (2026)
- [AgilePE: Autonomous UAV Pursuit-Evasion via Self-Play Reinforcement Learning](https://mlsanity.com/paper/2608.14135/) (2026)
- [Statistical Turbulence and High-Fidelity Disturbance Fields for Quadrotor Flight Control](https://mlsanity.com/paper/2610.06874/) (2026)

## Abstract (arXiv)

Executing precise and agile flight maneuvers is critical for quadrotors in various applications. Traditional quadrotor control approaches are limited by their reliance on flat trajectories or time-consuming optimization, which restricts their flexibility. Recently, RL-based policy has emerged as a promising alternative due to its ability to directly map observations to actions, reducing the need for detailed system knowledge and actuation constraints. However, a significant challenge remains in bridging the sim-to-real gap, where RL-based policies often experience instability when deployed in real world. In this paper, we investigate key factors for learning robust RL-based control policies that are capable of zero-shot deployment in real-world quadrotors. We identify five critical factors and we develop a PPO-based training framework named SimpleFlight, which integrates these five techniques. We validate the efficacy of SimpleFlight on Crazyflie quadrotor, demonstrating that it achieves more than a 50% reduction in trajectory tracking error compared to state-of-the-art RL baselines. The policy derived by SimpleFlight consistently excels across both smooth polynominal trajectories and challenging infeasible zigzag trajectories on small thrust-to-weight quadrotors. In contrast, baseline methods struggle with high-speed or infeasible trajectories. To support further research and reproducibility, we integrate SimpleFlight into a GPU-based simulator Omnidrones and provide open-source access to the code and model checkpoints. We hope SimpleFlight will offer valuable insights for advancing RL-based quadrotor control. For more details, visit our project website at https://sites.google.com/view/simpleflight/.