# Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics

arXiv: https://arxiv.org/abs/2407.10648 · HTML: https://mlsanity.com/paper/2407.10648/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2024.

## Built on by (most cited first, 30 of 45)

- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [Learning Quadrotor Control From Visual Features Using Differentiable Simulation](https://mlsanity.com/paper/2410.15979/) (2024)
- [Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation](https://mlsanity.com/paper/2410.03076/) (2024)
- [SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2510.14783/) (2025)
- [Aerial Vision-Language Navigation with a Unified Framework for Spatial, Temporal and Embodied Reasoning](https://mlsanity.com/paper/2512.08639/) (2025)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [Flying on Point Clouds with Reinforcement Learning](https://mlsanity.com/paper/2503.00496/) (2025)
- [Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.16720/) (2024)
- [RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation](https://mlsanity.com/paper/2502.02054/) (2025)
- [Quadrotor Navigation using Reinforcement Learning with Privileged Information](https://mlsanity.com/paper/2509.08177/) (2025)
- [Reactive Aerobatic Flight via Reinforcement Learning](https://mlsanity.com/paper/2505.24396/) (2025)
- [Reinforcement Learning for Active Perception in Autonomous Navigation](https://mlsanity.com/paper/2602.01266/) (2026)
- [Reparameterization Flow Policy Optimization](https://mlsanity.com/paper/2602.03501/) (2026)
- [Depth Transfer: Learning to See Like a Simulator for Real-World Drone Navigation](https://mlsanity.com/paper/2505.12428/) (2025)
- [DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning](https://mlsanity.com/paper/2509.10247/) (2025)
- [Mastering Diverse, Unknown, and Cluttered Tracks for Robust Vision-Based Drone Racing](https://mlsanity.com/paper/2512.09571/) (2025)
- [DiffSim2Real: Deploying Quadrupedal Locomotion Policies Purely Trained in Differentiable Simulation](https://mlsanity.com/paper/2411.02189/) (2024)
- [Fly360: Omnidirectional Obstacle Avoidance within Drone View](https://mlsanity.com/paper/2603.06573/) (2026)
- [Multi-Robot Collaboration through Reinforcement Learning and Abstract Simulation](https://mlsanity.com/paper/2503.05092/) (2025)
- [Towards Task-Oriented Flying: Framework, Infrastructure, and Principles](https://mlsanity.com/paper/2504.15129/) (2025)
- [Reparameterization Proximal Policy Optimization](https://mlsanity.com/paper/2508.06214/) (2025)
- [Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies](https://mlsanity.com/paper/2510.05692/) (2025)
- [Efficient Model-Based Reinforcement Learning for Robot Control via Online Optimization](https://mlsanity.com/paper/2510.18518/) (2025)
- [High-Speed Vision-Based Flight in Clutter with Safety-Shielded Reinforcement Learning](https://mlsanity.com/paper/2602.08653/) (2026)
- [Curriculum Reinforcement Learning for Quadrotor Racing with Random Obstacles](https://mlsanity.com/paper/2602.24030/) (2026)
- [Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environments](https://mlsanity.com/paper/2603.07578/) (2026)
- [ViBA: Implicit Bundle Adjustment with Geometric and Temporal Consistency for Robust Visual Matching](https://mlsanity.com/paper/2604.03377/) (2026)
- [NavRL++: A System-Level Framework for Improving Sim-to-Real Transfer in Reinforcement Learning-Based Robot Navigation](https://mlsanity.com/paper/2605.15559/) (2026)
- [Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient](https://mlsanity.com/paper/2605.26478/) (2026)
- [Crazyflow: An Accurate, GPU-Accelerated, Differentiable Drone Simulator in JAX](https://mlsanity.com/paper/2606.01478/) (2026)

## Abstract (arXiv)

Swarm navigation in cluttered environments is a grand challenge in robotics. This work combines deep learning with first-principle physics through differentiable simulation to enable autonomous navigation of multiple aerial robots through complex environments at high speed. Our approach optimizes a neural network control policy directly by backpropagating loss gradients through the robot simulation using a simple point-mass physics model and a depth rendering engine. Despite this simplicity, our method excels in challenging tasks for both multi-agent and single-agent applications with zero-shot sim-to-real transfer. In multi-agent scenarios, our system demonstrates self-organized behavior, enabling autonomous coordination without communication or centralized planning - an achievement not seen in existing traditional or learning-based methods. In single-agent scenarios, our system achieves a 90% success rate in navigating through complex environments, significantly surpassing the 60% success rate of the previous state-of-the-art approach. Our system can operate without state estimation and adapt to dynamic obstacles. In real-world forest environments, it navigates at speeds up to 20 m/s, doubling the speed of previous imitation learning-based solutions. Notably, all these capabilities are deployed on a budget-friendly 21 computer, costing less than 5% of a GPU-equipped board used in existing systems. Video demonstrations are available at https://youtu.be/LKg9hJqc2cc.