# Policy Search for Model Predictive Control with Application to Agile Drone Flight

arXiv: https://arxiv.org/abs/2112.03850 · HTML: https://mlsanity.com/paper/2112.03850/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 79 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 29 of 29)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving](https://mlsanity.com/paper/1707.02342/) (2017)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Deep Drone Acrobatics](https://mlsanity.com/paper/2006.05768/) (2020)
- [Towards Generalization and Simplicity in Continuous Control](https://mlsanity.com/paper/1703.02660/) (2017)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning](https://mlsanity.com/paper/2103.14666/) (2021)
- [Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search](https://mlsanity.com/paper/1509.06791/) (2015)
- [Robust and Efficient Quadrotor Trajectory Generation for Fast Autonomous Flight](https://mlsanity.com/paper/1907.01531/) (2019)
- [Enforcing robust control guarantees within neural network policies](https://mlsanity.com/paper/2011.08105/) (2020)
- [A survey on policy search algorithms for learning robot controllers in a handful of trials](https://mlsanity.com/paper/1807.02303/) (2018)
- [AlphaPilot: Autonomous Drone Racing](https://mlsanity.com/paper/2005.12813/) (2020)
- [PAMPC: Perception-Aware Model Predictive Control for Quadrotors](https://mlsanity.com/paper/1804.04811/) (2018)
- [RAPTOR: Robust and Perception-aware Trajectory Replanning for Quadrotor Fast Flight](https://mlsanity.com/paper/2007.03465/) (2020)
- [Time-Optimal Planning for Quadrotor Waypoint Flight](https://mlsanity.com/paper/2108.04537/) (2021)
- [Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing](https://mlsanity.com/paper/1810.06224/) (2018)
- [Search-based Motion Planning for Aggressive Flight in SE(3)](https://mlsanity.com/paper/1710.02748/) (2017)
- [Learning How to Autonomously Race a Car: a Predictive Control Approach](https://mlsanity.com/paper/1901.08184/) (2019)
- [Learning High-Level Policies for Model Predictive Control](https://mlsanity.com/paper/2007.10284/) (2020)
- [Model Predictive Control for Micro Aerial Vehicles: A Survey](https://mlsanity.com/paper/2011.11104/) (2020)
- [Vision-Based High Speed Driving with a Deep Dynamic Observer](https://mlsanity.com/paper/1812.02071/) (2018)

## Built on by (most cited first, 20 of 20)

- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [Learning to Warm-Start Fixed-Point Optimization Algorithms](https://mlsanity.com/paper/2309.07835/) (2023)
- [Learning Perception-Aware Agile Flight in Cluttered Environments](https://mlsanity.com/paper/2210.01841/) (2022)
- [Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomy](https://mlsanity.com/paper/2406.16087/) (2024)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Learning Speed Adaptation for Flight in Clutter](https://mlsanity.com/paper/2403.04586/) (2024)
- [Vision-based Learning for Drones: A Survey](https://mlsanity.com/paper/2312.05019/) (2023)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Learning-Aided Warmstart of Model Predictive Control in Uncertain Fast-Changing Traffic](https://mlsanity.com/paper/2310.02918/) (2023)
- [Bilevel Multi-Armed Bandit-Based Hierarchical Reinforcement Learning for Interaction-Aware Self-Driving at Unsignalized Intersections](https://mlsanity.com/paper/2502.03960/) (2025)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [Bi-Level Optimization Augmented with Conditional Variational Autoencoder for Autonomous Driving in Dense Traffic](https://mlsanity.com/paper/2212.02224/) (2022)
- [Learning the References of Online Model Predictive Control for Urban Self-Driving](https://mlsanity.com/paper/2308.15808/) (2023)
- [Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC](https://mlsanity.com/paper/2412.19669/) (2024)
- [Learning Agile Gate Traversal via Analytical Optimal Policy Gradient](https://mlsanity.com/paper/2508.21592/) (2025)
- [Agile Robotics: Optimal Control, Reinforcement Learning, and Differentiable Simulation](https://mlsanity.com/paper/2407.01568/) (2024)
- [Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategies](https://mlsanity.com/paper/2606.06011/) (2026)
- [HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization](https://mlsanity.com/paper/2606.16480/) (2026)
- [Distributed Secure Learning Control for Large-scale Multirobots under Stealthy Actuator Attacks](https://mlsanity.com/paper/2609.06896/) (2026)

## Abstract (arXiv)

Policy Search and Model Predictive Control~(MPC) are two different paradigms for robot control: policy search has the strength of automatically learning complex policies using experienced data, while MPC can offer optimal control performance using models and trajectory optimization. An open research question is how to leverage and combine the advantages of both approaches. In this work, we provide an answer by using policy search for automatically choosing high-level decision variables for MPC, which leads to a novel policy-search-for-model-predictive-control framework. Specifically, we formulate the MPC as a parameterized controller, where the hard-to-optimize decision variables are represented as high-level policies. Such a formulation allows optimizing policies in a self-supervised fashion. We validate this framework by focusing on a challenging problem in agile drone flight: flying a quadrotor through fast-moving gates. Experiments show that our controller achieves robust and real-time control performance in both simulation and the real world. The proposed framework offers a new perspective for merging learning and control.