# Model-predictive control and reinforcement learning in multi-energy system case studies

arXiv: https://arxiv.org/abs/2104.09785 · HTML: https://mlsanity.com/paper/2104.09785/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 79 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [From Reinforcement Learning to Optimal Control: A unified framework for sequential decisions](https://mlsanity.com/paper/1912.03513/) (2019)
- [ModelicaGym: Applying Reinforcement Learning to Modelica Models](https://mlsanity.com/paper/1909.08604/) (2019)
- [Weighted Random Search for Hyperparameter Optimization](https://mlsanity.com/paper/2004.01628/) (2020)
- [Deep Reinforcement Learning based Optimal Control of Hot Water Systems](https://mlsanity.com/paper/1801.01467/) (2018)
- [Reinforcement Learning for the Unit Commitment Problem](https://mlsanity.com/paper/1507.05268/) (2015)
- [Model-Free Control of Thermostatically Controlled Loads Connected to a District Heating Network](https://mlsanity.com/paper/1701.08074/) (2017)

## Built on by (most cited first, 10 of 10)

- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Physics Informed Neural Networks for Control Oriented Thermal Modeling of Buildings](https://mlsanity.com/paper/2111.12066/) (2021)
- [Safe reinforcement learning for multi-energy management systems with known constraint functions](https://mlsanity.com/paper/2207.03830/) (2022)
- [An adaptive safety layer with hard constraints for safe reinforcement learning in multi-energy management systems](https://mlsanity.com/paper/2304.08897/) (2023)
- [Wasserstein Auto-encoded MDPs: Formal Verification of Efficiently Distilled RL Policies with Many-sided Guarantees](https://mlsanity.com/paper/2303.12558/) (2023)
- [TreeC: a method to generate interpretable energy management systems using a metaheuristic algorithm](https://mlsanity.com/paper/2304.08310/) (2023)
- [Generalized Policy Learning for Smart Grids: FL TRPO Approach](https://mlsanity.com/paper/2403.18439/) (2024)
- [Real-world validation of safe reinforcement learning, model predictive control and decision tree-based home energy management systems](https://mlsanity.com/paper/2408.07435/) (2024)
- [Generalising Battery Control in Net-Zero Buildings via Personalised Federated RL](https://mlsanity.com/paper/2412.20946/) (2024)
- [A Statistical and Machine Learning Framework for Operational Threshold Detection and Deployable Dispatch Controller Development in Hydrogen Multi-Energy Systems](https://mlsanity.com/paper/2606.14601/) (2026)

## Abstract (arXiv)

Model-predictive-control (MPC) offers an optimal control technique to establish and ensure that the total operation cost of multi-energy systems remains at a minimum while fulfilling all system constraints. However, this method presumes an adequate model of the underlying system dynamics, which is prone to modelling errors and is not necessarily adaptive. This has an associated initial and ongoing project-specific engineering cost. In this paper, we present an on- and off-policy multi-objective reinforcement learning (RL) approach, that does not assume a model a priori, benchmarking this against a linear MPC (LMPC - to reflect current practice, though non-linear MPC performs better) - both derived from the general optimal control problem, highlighting their differences and similarities. In a simple multi-energy system (MES) configuration case study, we show that a twin delayed deep deterministic policy gradient (TD3) RL agent offers potential to match and outperform the perfect foresight LMPC benchmark (101.5%). This while the realistic LMPC, i.e. imperfect predictions, only achieves 98%. While in a more complex MES system configuration, the RL agent's performance is generally lower (94.6%), yet still better than the realistic LMPC (88.9%). In both case studies, the RL agents outperformed the realistic LMPC after a training period of 2 years using quarterly interactions with the environment. We conclude that reinforcement learning is a viable optimal control technique for multi-energy systems given adequate constraint handling and pre-training, to avoid unsafe interactions and long training periods, as is proposed in fundamental future work.