# Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight

arXiv: https://arxiv.org/abs/2306.09852 · HTML: https://mlsanity.com/paper/2306.09852/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2023.

## Built on by (most cited first, 19 of 19)

- [Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight](https://mlsanity.com/paper/2501.14377/) (2025)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking](https://mlsanity.com/paper/2510.02490/) (2025)
- [DiffCoord: Differentiable Coordination for Distributed Multi-Agent Trajectory Optimization](https://mlsanity.com/paper/2509.01630/) (2025)
- [Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift](https://mlsanity.com/paper/2604.01142/) (2026)
- [Learning Agile Gate Traversal via Analytical Optimal Policy Gradient](https://mlsanity.com/paper/2508.21592/) (2025)
- [Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making](https://mlsanity.com/paper/2512.17091/) (2025)
- [Inference Time Policy Optimization for Offline RL with Differentiable World Models](https://mlsanity.com/paper/2603.22430/) (2026)
- [Mahalanobis-Guided Latent OOD Detection for Hybrid ES-DRL Control in Time-Varying Systems](https://mlsanity.com/paper/2606.11474/) (2026)
- [Dynamics-Invariant Quadrotor Control using Scale-Aware Deep Reinforcement Learning](https://mlsanity.com/paper/2503.09622/) (2025)
- [Q-Guided Stein Variational Model Predictive Control via RL-informed Policy Prior](https://mlsanity.com/paper/2507.06625/) (2025)
- [RAY-TOLD: Ray-Based Latent Dynamics for Dense Dynamic Obstacle Avoidance with TDMPC](https://mlsanity.com/paper/2604.27450/) (2026)
- [CA-AC-MPC: CUDA-Accelerated Actor-Critic Model Predictive Control](https://mlsanity.com/paper/2605.29155/) (2026)
- [Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategies](https://mlsanity.com/paper/2606.06011/) (2026)
- [SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control](https://mlsanity.com/paper/2608.21175/) (2026)
- [Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC](https://mlsanity.com/paper/2609.01061/) (2026)
- [Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic](https://mlsanity.com/paper/2609.30696/) (2026)
- [RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments](https://mlsanity.com/paper/2609.39854/) (2026)
- [Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning](https://mlsanity.com/paper/2610.06019/) (2026)

## Abstract (arXiv)

A key open challenge in agile quadrotor flight is how to combine the flexibility and task-level generality of model-free reinforcement learning (RL) with the structure and online replanning capabilities of model predictive control (MPC), aiming to leverage their complementary strengths in dynamic and uncertain environments. This paper provides an answer by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an actor-critic RL framework. This integration allows for short-term predictive optimization of control actions through MPC, while leveraging RL for end-to-end learning and exploration over longer horizons. Through various ablation studies, conducted in the context of agile quadrotor racing, we expose the benefits of the proposed approach: it achieves better out-of-distribution behaviour, better robustness to changes in the quadrotor's dynamics and improved sample efficiency. Additionally, we conduct an empirical analysis using a quadrotor platform that reveals a relationship between the critic's learned value function and the cost function of the differentiable MPC, providing a deeper understanding of the interplay between the critic's value and the MPC cost functions. Finally, we validate our method in a drone racing task on different tracks, in both simulation and the real world. Our method achieves the same superhuman performance as state-of-the-art model-free RL, showcasing speeds of up to 21 m/s. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the predictive properties of the MPC to better handle out-of-distribution behavior.