# Learning Quadrotor Control From Visual Features Using Differentiable Simulation

arXiv: https://arxiv.org/abs/2410.15979 · HTML: https://mlsanity.com/paper/2410.15979/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Gradients are Not All You Need](https://mlsanity.com/paper/2111.05803/) (2021)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [The Double Sphere Camera Model](https://mlsanity.com/paper/1807.08957/) (2018)
- [Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics](https://mlsanity.com/paper/2407.10648/) (2024)
- [Learning Quadruped Locomotion Using Differentiable Simulation](https://mlsanity.com/paper/2403.14864/) (2024)
- [Learning to Fly in Seconds](https://mlsanity.com/paper/2311.13081/) (2023)
- [Demonstrating Agile Flight from Pixels without State Estimation](https://mlsanity.com/paper/2406.12505/) (2024)
- [AlphaPilot: Autonomous Drone Racing](https://mlsanity.com/paper/2005.12813/) (2020)
- [PAMPC: Perception-Aware Model Predictive Control for Quadrotors](https://mlsanity.com/paper/1804.04811/) (2018)
- [A Comparative Study of Nonlinear MPC and Differential-Flatness-Based Control for Quadrotor Agile Flight](https://mlsanity.com/paper/2109.01365/) (2021)
- [Linear vs Nonlinear MPC for Trajectory Tracking Applied to Rotary Wing Micro Aerial Vehicles](https://mlsanity.com/paper/1611.09240/) (2016)

## Built on by (most cited first, 19 of 19)

- [RAPTOR: A Foundation Policy for Quadrotor Control](https://mlsanity.com/paper/2509.11481/) (2025)
- [Flying on Point Clouds with Reinforcement Learning](https://mlsanity.com/paper/2503.00496/) (2025)
- [Quadrotor Navigation using Reinforcement Learning with Privileged Information](https://mlsanity.com/paper/2509.08177/) (2025)
- [Accelerating Visual-Policy Learning through Parallel Differentiable Simulation](https://mlsanity.com/paper/2505.10646/) (2025)
- [Reparameterization Flow Policy Optimization](https://mlsanity.com/paper/2602.03501/) (2026)
- [DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning](https://mlsanity.com/paper/2509.10247/) (2025)
- [Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Models](https://mlsanity.com/paper/2502.10012/) (2025)
- [Towards Task-Oriented Flying: Framework, Infrastructure, and Principles](https://mlsanity.com/paper/2504.15129/) (2025)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [Autonomous Vehicle Path Planning by Searching With Differentiable Simulation](https://mlsanity.com/paper/2511.11043/) (2025)
- [Agile Interception of a Flying Target using Competitive Reinforcement Learning](https://mlsanity.com/paper/2603.16279/) (2026)
- [Crazyflow: An Accurate, GPU-Accelerated, Differentiable Drone Simulator in JAX](https://mlsanity.com/paper/2606.01478/) (2026)
- [An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer](https://mlsanity.com/paper/2503.10118/) (2025)
- [VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control](https://mlsanity.com/paper/2603.21123/) (2026)
- [Agile Fall Recovery for Quadrotors with Bidirectional Thrust via Reinforcement Learning](https://mlsanity.com/paper/2606.16513/) (2026)
- [Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control](https://mlsanity.com/paper/2609.20575/) (2026)
- [Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic](https://mlsanity.com/paper/2609.30696/) (2026)
- [When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observability](https://mlsanity.com/paper/2609.33169/) (2026)
- [ClearNav: Perception-Aligned Quadrotor Flight Using Differential-Physics](https://mlsanity.com/paper/2610.17419/) (2026)

## Abstract (arXiv)

The sample inefficiency of reinforcement learning (RL) remains a significant challenge in robotics. RL requires large-scale simulation and can still cause long training times, slowing research and innovation. This issue is particularly pronounced in vision-based control tasks where reliable state estimates are not accessible. Differentiable simulation offers an alternative by enabling gradient back-propagation through the dynamics model, providing low-variance analytical policy gradients and, hence, higher sample efficiency. However, its usage for real-world robotic tasks has yet been limited. This work demonstrates the great potential of differentiable simulation for learning quadrotor control. We show that training in differentiable simulation significantly outperforms model-free RL in terms of both sample efficiency and training time, allowing a policy to learn to recover a quadrotor in seconds when providing vehicle states and in minutes when relying solely on visual features. The key to our success is two-fold. First, the use of a simple surrogate model for gradient computation greatly accelerates training without sacrificing control performance. Second, combining state representation learning with policy learning enhances convergence speed in tasks where only visual features are observable. These findings highlight the potential of differentiable simulation for real-world robotics and offer a compelling alternative to conventional RL approaches.