# VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control

arXiv: https://arxiv.org/abs/2603.21123 · HTML: https://mlsanity.com/paper/2603.21123/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 22 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [NeuralSim: Augmenting Differentiable Simulators with Neural Networks](https://mlsanity.com/paper/2011.04217/) (2020)
- [Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics](https://mlsanity.com/paper/2407.10648/) (2024)
- [Dojo: A Differentiable Physics Engine for Robotics](https://mlsanity.com/paper/2203.00806/) (2022)
- [Training Efficient Controllers via Analytic Policy Gradient](https://mlsanity.com/paper/2209.13052/) (2022)
- [Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation](https://mlsanity.com/paper/2405.17784/) (2024)
- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [Learning Quadrotor Control From Visual Features Using Differentiable Simulation](https://mlsanity.com/paper/2410.15979/) (2024)
- [Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation](https://mlsanity.com/paper/2508.21065/) (2025)
- [VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight](https://mlsanity.com/paper/2407.14783/) (2024)
- [YOPOv2-Tracker: An End-to-End Agile Tracking and Navigation Framework from Perception to Action](https://mlsanity.com/paper/2505.06923/) (2025)

## Abstract (arXiv)

First-order reinforcement learning with differentiable simulation is promising for quadrotor control, but practical progress remains fragmented across task-specific settings. To support more systematic development and evaluation, we present a unified differentiable framework for multi-task quadrotor control. The framework is wrapped, extensible, and equipped with deployment-oriented dynamics, providing a common interface across four representative tasks: hovering, tracking, landing, and racing. We also present the suite of first-order learning algorithms, where we identify two practical bottlenecks of standard first-order training: limited state coverage caused by horizon initialization and gradient bias caused by partially non-differentiable rewards. To address these issues, we propose Amended Backpropagation Through Time (ABPT), which combines differentiable rollout optimization, a value-based auxiliary objective, and visited-state initialization to improve training robustness. Experimental results show that ABPT yields the clearest gains in tasks with partially non-differentiable rewards, while remaining competitive in fully differentiable settings. We further provide proof-of-concept real-world deployments showing initial transferability of policies learned in the proposed framework beyond simulation.