mlsanityData to 14 Oct 2026

Learning Acrobatic Flight from Preferences

arXiv 2508.18817 ↗First posted 26 Aug 2025cs.RO cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot navigation and UAV control

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) enables agents to learn control policies without requiring manually designed reward functions, making it well-suited for tasks where objectives are difficult to formalize or inherently subjective. Acrobatic flight poses a particularly challenging problem due to its complex dynamics, rapid movements, and the importance of precise execution. However, manually designed reward functions for such tasks often fail to capture the qualities that matter: we find that hand-crafted rewards agree with human judgment only 60.7% of the time, underscoring the need for preference-driven approaches. In this work, we propose Reward Ensemble under Confidence (REC), a probabilistic reward learning framework for PbRL that explicitly models per-timestep reward uncertainty through an ensemble of distributional reward models. By propagating uncertainty into the preference loss and leveraging disagreement for exploration, REC achieves 88.4% of shaped reward performance on acrobatic quadrotor control, compared to 55.2% with standard Preference PPO. We train policies in simulation and successfully transfer them zero-shot to the real world, demonstrating complex acrobatic maneuvers learned purely from preference feedback. We further validate REC on a continuous control benchmark, confirming its applicability beyond the domain of aerial robotics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 29 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

21 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Multi-Task Reinforcement Learning for Quadrotors17 Dec
  2. 2024Demonstrating Agile Flight from Pixels without State Estimation18 Jun
  3. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight18 Mar

Other references · 18

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  5. 2023Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learningcited by 111
  6. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  7. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  8. 2020Flightmare: A Flexible Quadrotor Simulatorcited by 84
  9. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  10. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 68
  11. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  12. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
6 more
  1. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  2. 2021NeuroBEM: Hybrid Aerodynamic Quadrotor Modelcited by 37
  3. 2023Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flightcited by 30
  4. 2023Autonomous Power Line Inspection with Drones via Perception-Aware MPCcited by 17
  5. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  6. 2025Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  2. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences
  3. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback
  4. 2025Reactive Aerobatic Flight via Reinforcement Learning
  5. 2025Leveling the Playing Field: Carefully Comparing Classical and Learned Controllers for Quadrotor Trajectory Tracking
  6. 2026VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control
  7. 2025Best Policy Learning from Trajectory Preference Feedback
  8. 2026A Heuristic Approach for Performance Tuning in RL-based Quadrotor Control via Reward Design and Termination Conditions