# Multi-Fidelity Reinforcement Learning for Time-Optimal Quadrotor Re-planning

arXiv: https://arxiv.org/abs/2403.08152 · HTML: https://mlsanity.com/paper/2403.08152/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 48 of its 83 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 30 of 48)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Manifold Gaussian Processes for Regression](https://mlsanity.com/paper/1402.5876/) (2014)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [Estimating Risk and Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/1905.09638/) (2019)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Deep Gaussian Processes for Multi-fidelity Modeling](https://mlsanity.com/paper/1903.07320/) (2019)
- [Sim2Real Transfer for Reinforcement Learning without Dynamics Randomization](https://mlsanity.com/paper/2002.11635/) (2020)
- [Feedback Control For Cassie With Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05580/) (2018)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)

## Built on by (most cited first, 4 of 4)

- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation](https://mlsanity.com/paper/2503.05696/) (2025)
- [Real-Time Sampling-based Online Planning for Drone Interception](https://mlsanity.com/paper/2502.14231/) (2025)
- [Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning](https://mlsanity.com/paper/2610.02505/) (2026)

## Abstract (arXiv)

High-speed online trajectory planning for UAVs poses a significant challenge due to the need for precise modeling of complex dynamics while also being constrained by computational limitations. This paper presents a multi-fidelity reinforcement learning method (MFRL) that aims to effectively create a realistic dynamics model and simultaneously train a planning policy that can be readily deployed in real-time applications. The proposed method involves the co-training of a planning policy and a reward estimator; the latter predicts the performance of the policy's output and is trained efficiently through multi-fidelity Bayesian optimization. This optimization approach models the correlation between different fidelity levels, thereby constructing a high-fidelity model based on a low-fidelity foundation, which enables the accurate development of the reward model with limited high-fidelity experiments. The framework is further extended to include real-world flight experiments in reinforcement learning training, allowing the reward model to precisely reflect real-world constraints and broadening the policy's applicability to real-world scenarios. We present rigorous evaluations by training and testing the planning policy in both simulated and real-world environments. The resulting trained policy not only generates faster and more reliable trajectories compared to the baseline snap minimization method, but it also achieves trajectory updates in 2 ms on average, while the baseline method takes several minutes.