# Bridging RL and MPC for mixed-integer optimal control with application to Formula 1 race strategies

arXiv: https://arxiv.org/abs/2604.00826 · HTML: https://mlsanity.com/paper/2604.00826/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 1 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Discrete and Continuous Action Representation for Practical RL in Video Games](https://mlsanity.com/paper/1912.11077/) (2019)
- [MPC-Net: A First Principles Guided Policy Search](https://mlsanity.com/paper/1909.05197/) (2019)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control](https://mlsanity.com/paper/2108.10315/) (2021)
- [Deep Value Model Predictive Control](https://mlsanity.com/paper/1910.03358/) (2019)
- [Imitation Learning from MPC for Quadrupedal Multi-Gait Control](https://mlsanity.com/paper/2103.14331/) (2021)
- [Reinforcement Learning for Mixed-Integer Problems Based on MPC](https://mlsanity.com/paper/2004.01430/) (2020)
- [A view on learning robust goal-conditioned value functions: Interplay between RL and MPC](https://mlsanity.com/paper/2502.06996/) (2025)
- [Towards Learning-Based Formula 1 Race Strategies](https://mlsanity.com/paper/2512.21570/) (2025)
- [Learning-Based MPC for Fuel Efficient Control of Autonomous Vehicles with Discrete Gear Selection](https://mlsanity.com/paper/2503.11359/) (2025)

## Abstract (arXiv)

We propose a hybrid reinforcement learning (RL) and model predictive control (MPC) framework for mixed-integer optimal control, where discrete variables enter the cost and dynamics but not the constraints. Existing hierarchical approaches learn a policy only for the discrete action space, leaving continuous optimization to MPC. Unlike these methods, we train an RL agent on the full hybrid action space, such that the learned critic approximates the Q-function of the underlying Markov decision process. During deployment, the RL actor is rolled out over the prediction horizon to parametrize an integer-free nonlinear MPC through the discrete action sequence and provide a continuous warm-start. The learned critic can additionally serve as a terminal cost to capture long-term performance. We prove recursive feasibility, and validate the framework on a Formula 1 race strategy problem, where an ablation study identifies the contribution of each learned component. The hybrid method achieves near-optimal performance relative to an offline mixed-integer nonlinear program benchmark, outperforming a standalone RL agent. Moreover, the hybrid scheme enables adaptation to unseen disturbances through modular MPC extensions at zero retraining cost.