mlsanityData to 14 Oct 2026

Demonstrating Agile Flight from Pixels without State Estimation

arXiv 2406.12505 ↗First posted 18 Jun 2024cs.ROcited by 31

On the map: Deep reinforcement learning methods → RL for robot navigation and UAV control

Abstract (arXiv)

Quadrotors are among the most agile flying robots. Despite recent advances in learning-based control and computer vision, autonomous drones still rely on explicit state estimation. On the other hand, human pilots only rely on a first-person-view video stream from the drone onboard camera to push the platform to its limits and fly robustly in unseen environments. To the best of our knowledge, we present the first vision-based quadrotor system that autonomously navigates through a sequence of gates at high speeds while directly mapping pixels to control commands. Like professional drone-racing pilots, our system does not use explicit state estimation and leverages the same control commands humans use (collective thrust and body rates). We demonstrate agile flight at speeds up to 40km/h with accelerations up to 2g. This is achieved by training vision-based policies with reinforcement learning (RL). The training is facilitated using an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use the inner edges of the gates as a sensor abstraction. This simple yet robust, task-relevant representation can be simulated during training without rendering images. During deployment, a Swin-transformer-based gate detector is used. Our approach enables autonomous agile flight with standard, off-the-shelf hardware. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 63 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

46 of 63 references.

References · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  8. 2018DeepMind Control Suitecited by 956
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  11. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  12. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
34 more
  1. 2020Reinforcement Learning with Augmented Datacited by 468
  2. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  3. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  4. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  6. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  7. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  8. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  9. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  10. 2022Deep Patch Visual Odometrycited by 190
  11. 2017DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networkscited by 175
  12. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  13. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  14. 2021Learning High-Speed Flight in the Wildcited by 128
  15. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 121
  16. 2023Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learningcited by 111
  17. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 94
  18. 2020Deep Drone Acrobaticscited by 75
  19. 2018The Double Sphere Camera Modelcited by 52
  20. 2018Reinforcement Learning for UAV Attitude Controlcited by 48
  21. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 48
  22. 2022A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flightcited by 47
  23. 2019Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learningcited by 44
  24. 2021NeuroBEM: Hybrid Aerodynamic Quadrotor Modelcited by 37
  25. 2020Self-Supervised Deep Visual Odometry with Online Adaptationcited by 35
  26. 2018Deep Drone Racing: Learning Agile Flight in Dynamic Environmentscited by 34
  27. 2023Learning to Fly in Secondscited by 31
  28. 2020AlphaPilot: Autonomous Drone Racingcited by 30
  29. 2023Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flightcited by 30
  30. 2017Fast, Autonomous Flight in GPS-Denied and Cluttered Environmentscited by 18
  31. 2023End-to-end Reinforcement Learning for Time-Optimal Quadcopter Flightcited by 17
  32. 2021Human-Piloted Drone Racing: Visual Processing and Controlcited by 9
  33. 2021Autonomous Flights in Dynamic Environments with Onboard Visioncited by 8
  34. 2016Aggressive Quadrotor Flight through Narrow Gaps with Onboard Sensing and Computing using Active Visioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 10.

31 papers.

Sort:

By area:

  1. 2024Multi-Task Reinforcement Learning for Quadrotorscited by 20
  2. 2024Learning Quadrotor Control From Visual Features Using Differentiable Simulationcited by 19
  3. 2024SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuumcited by 16
  4. 2025Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flightcited by 15
  5. 2024Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulationcited by 13
  6. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Modelscited by 10
  7. 2025SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learningcited by 8
  8. 2025Flying on Point Clouds with Reinforcement Learningcited by 7
  9. 2025One Net to Rule Them All: Domain Randomization in Quadcopter Racing Across Different Platformscited by 7
  10. 2024Environment as Policy: Learning to Race in Unseen Trackscited by 6
  11. 2025RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigationcited by 6
  12. 2024Benchmarking Deep Reinforcement Learning for Navigation in Denied Sensor Environmentscited by 4
19 more
  1. 2025Multi-Robot Collaboration through Reinforcement Learning and Abstract Simulationcited by 1
  2. 2025Neural Fidelity Calibration for Informative Sim-to-Real Adaptationcited by 1
  3. 2025All Eyes, no IMU: Learning Flight Attitude from Vision Alonecited by 1
  4. 2025Synthetic vs. Real Training Data for Visual Navigationcited by 1
  5. 2025Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalizationcited by 1
  6. 2026Curriculum Reinforcement Learning for Quadrotor Racing with Random Obstaclescited by 1
  7. 2026Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environmentscited by 1
  8. 2026Superhuman Safe and Agile Racing through Multi-Agent Reinforcement Learningcited by 1
  9. 2025A Reinforcement Learning Approach to Non-prehensile Manipulation through Slidingcited by 0
  10. 2025AttentionSwarm: Reinforcement Learning with Attention Control Barier Function for Crazyflie Drones in Dynamic Environmentscited by 0
  11. 2025Self-Supervised Monocular Visual Drone Model Identification through Improved Occlusion Handlingcited by 0
  12. 2025Learning Acrobatic Flight from Preferencescited by 0
  13. 2026Agile Fall Recovery for Quadrotors with Bidirectional Thrust via Reinforcement Learningcited by 0
  14. 2026Bridging Performance and Generalization in Reinforcement Learning for Agile Flightcited by 0
  15. 2026FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Trackingcited by 0
  16. 2026Toward the Cognitive--Physical Limits of Embodied Intelligence through a World-Model-Centric Autonomous Racing Agentcited by 0
  17. 2026Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Criticcited by 0
  18. 2026Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasioncited by 0
  19. 2026ClearNav: Perception-Aligned Quadrotor Flight Using Differential-Physicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight
  2. 2022Learning Perception-Aware Agile Flight in Cluttered Environments
  3. 2026MAD: Mapping-Aware World Models for Agile Quadrotor Flight
  4. 2025Mastering Diverse, Unknown, and Cluttered Tracks for Robust Vision-Based Drone Racing
  5. 2025Reactive Aerobatic Flight via Reinforcement Learning
  6. 2026PerchRL: Vision-Based Agile Perching on Inclined Platforms under Rapid and Irregular Motion
  7. 2026MAVEN: A Meta-Reinforcement Learning Framework for Varying-Dynamics Expertise in Agile Quadrotor Maneuvers
  8. 2026Precise Aggressive Aerial Maneuvers with Sensorimotor Policies