# Integrating Reinforcement Learning and Model Predictive Control with Applications to Microgrids

arXiv: https://arxiv.org/abs/2409.11267 · HTML: https://mlsanity.com/paper/2409.11267/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Reinforcement Learning for Integer Programming: Learning to Cut](https://mlsanity.com/paper/1906.04859/) (2019)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Learning to Branch](https://mlsanity.com/paper/1803.10150/) (2018)
- [Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/1509.01644/) (2015)
- [Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1903.01344/) (2019)
- [Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics](https://mlsanity.com/paper/2001.00449/) (2020)
- [A Survey for Solving Mixed Integer Programming via Machine Learning](https://mlsanity.com/paper/2203.02878/) (2022)
- [CoCo: Online Mixed-Integer Control via Supervised Learning](https://mlsanity.com/paper/2107.08143/) (2021)
- [Learning for MPC with Stability & Safety Guarantees](https://mlsanity.com/paper/2012.07369/) (2020)
- [Reinforcement Learning for Mixed-Integer Problems Based on MPC](https://mlsanity.com/paper/2004.01430/) (2020)
- [Optimal Energy System Scheduling Using A Constraint-Aware Reinforcement Learning Algorithm](https://mlsanity.com/paper/2305.05484/) (2023)
- [Learning for Online Mixed-Integer Model Predictive Control with Parametric Optimality Certificates](https://mlsanity.com/paper/2303.12152/) (2023)
- [Learning Q-function approximations for hybrid control problems](https://mlsanity.com/paper/2105.13517/) (2021)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning with Distributed MPC for Fuel-Efficient Platoon Control with Discrete Gear Transitions](https://mlsanity.com/paper/2601.18294/) (2026)

## Abstract (arXiv)

This work proposes an approach that integrates reinforcement learning and model predictive control (MPC) to solve finite-horizon optimal control problems in mixed-logical dynamical systems efficiently. Optimization-based control of such systems with discrete and continuous decision variables entails the online solution of mixed-integer linear programs, which suffer from the curse of dimensionality. Our approach aims to mitigate this issue by decoupling the decision on the discrete variables from the decision on the continuous variables. In the proposed approach, reinforcement learning determines the discrete decision variables and simplifies the online optimization problem of the MPC controller from a mixed-integer linear program to a linear program, significantly reducing the computational time. A fundamental contribution of this work is the definition of the decoupled Q-function, which plays a crucial role in making the learning problem tractable in a combinatorial action space. We motivate the use of recurrent neural networks to approximate the decoupled Q-function and show how they can be employed in a reinforcement learning setting. Simulation experiments on a microgrid system using real-world data demonstrate that the proposed method substantially reduces the online computation time of MPC while maintaining high feasibility and low suboptimality.