# Safe reinforcement learning for multi-energy management systems with known constraint functions

arXiv: https://arxiv.org/abs/2207.03830 · HTML: https://mlsanity.com/paper/2207.03830/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.06655/) (2021)
- [Model-predictive control and reinforcement learning in multi-energy system case studies](https://mlsanity.com/paper/2104.09785/) (2021)
- [Real-world challenges for multi-agent reinforcement learning in grid-interactive buildings](https://mlsanity.com/paper/2112.06127/) (2021)
- [A Learning-based Power Management for Networked Microgrids Under Incomplete Information](https://mlsanity.com/paper/1810.01758/) (2018)
- [ModelicaGym: Applying Reinforcement Learning to Modelica Models](https://mlsanity.com/paper/1909.08604/) (2019)
- [Energy Management Based on Multi-Agent Deep Reinforcement Learning for A Multi-Energy Industrial Park](https://mlsanity.com/paper/2202.03771/) (2022)

## Built on by (most cited first, 7 of 7)

- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [Learning the Optimal Power Flow: Environment Design Matters](https://mlsanity.com/paper/2403.17831/) (2024)
- [An adaptive safety layer with hard constraints for safe reinforcement learning in multi-energy management systems](https://mlsanity.com/paper/2304.08897/) (2023)
- [A Critical Review of Safe Reinforcement Learning Techniques in Smart Grid Applications](https://mlsanity.com/paper/2409.16256/) (2024)
- [Constrained Online Decision-Making: A Unified Framework](https://mlsanity.com/paper/2505.07101/) (2025)
- [Real-world validation of safe reinforcement learning, model predictive control and decision tree-based home energy management systems](https://mlsanity.com/paper/2408.07435/) (2024)
- [Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgrids](https://mlsanity.com/paper/2512.01046/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) is a promising optimal control technique for multi-energy management systems. It does not require a model a priori - reducing the upfront and ongoing project-specific engineering effort and is capable of learning better representations of the underlying system dynamics. However, vanilla RL does not provide constraint satisfaction guarantees - resulting in various potentially unsafe interactions within its safety-critical environment. In this paper, we present two novel safe RL methods, namely SafeFallback and GiveSafe, where the safety constraint formulation is decoupled from the RL formulation. These provide hard-constraint, rather than soft- and chance-constraint, satisfaction guarantees both during training a (near) optimal policy (which involves exploratory and exploitative, i.e. greedy, steps) as well as during deployment of any policy (e.g. random agents or offline trained RL agents). This without the need of solving a mathematical program, resulting in less computational power requirements and a more flexible constraint function formulation (no derivative information is required). In a simulated multi-energy systems case study we have shown that both methods start with a significantly higher utility (i.e. useful policy) compared to a vanilla RL benchmark and Optlayer benchmark (94,6% and 82,8% compared to 35,5% and 77,8%) and that the proposed SafeFallback method even can outperform the vanilla RL benchmark (102,9% to 100%). We conclude that both methods are viably safety constraint handling techniques applicable beyond RL, as demonstrated with random policies while still providing hard-constraint guarantees.