# An adaptive safety layer with hard constraints for safe reinforcement learning in multi-energy management systems

arXiv: https://arxiv.org/abs/2304.08897 · HTML: https://mlsanity.com/paper/2304.08897/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Conservative Safety Critics for Exploration](https://mlsanity.com/paper/2010.14497/) (2020)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.06655/) (2021)
- [Model-predictive control and reinforcement learning in multi-energy system case studies](https://mlsanity.com/paper/2104.09785/) (2021)
- [Safe reinforcement learning for multi-energy management systems with known constraint functions](https://mlsanity.com/paper/2207.03830/) (2022)
- [Energy Management Based on Multi-Agent Deep Reinforcement Learning for A Multi-Energy Industrial Park](https://mlsanity.com/paper/2202.03771/) (2022)
- [Energy Trading in Microgrids for Synergies among Electricity, Hydrogen and Heat Networks](https://mlsanity.com/paper/2005.00416/) (2020)
- [Stochastic Gradient-based Fast Distributed Multi-Energy Management for an Industrial Park with Temporally-Coupled Constraints](https://mlsanity.com/paper/2204.04088/) (2022)

## Built on by (most cited first, 2 of 2)

- [Safe Reinforcement Learning for Power System Control: A Review](https://mlsanity.com/paper/2407.00681/) (2024)
- [Real-world validation of safe reinforcement learning, model predictive control and decision tree-based home energy management systems](https://mlsanity.com/paper/2408.07435/) (2024)

## Abstract (arXiv)

Safe reinforcement learning (RL) with hard constraint guarantees is a promising optimal control direction for multi-energy management systems. It only requires the environment-specific constraint functions itself a priori and not a complete model. The project-specific upfront and ongoing engineering efforts are therefore still reduced, better representations of the underlying system dynamics can still be learnt, and modelling bias is kept to a minimum. However, even the constraint functions alone are not always trivial to accurately provide in advance, leading to potentially unsafe behaviour. In this paper, we present two novel advancements: (I) combining the OptLayer and SafeFallback method, named OptLayerPolicy, to increase the initial utility while keeping a high sample efficiency and the possibility to formulate equality constraints. (II) introducing self-improving hard constraints, to increase the accuracy of the constraint functions as more and new data becomes available so that better policies can be learnt. Both advancements keep the constraint formulation decoupled from the RL formulation, so new (presumably better) RL algorithms can act as drop-in replacements. We have shown that, in a simulated multi-energy system case study, the initial utility is increased to 92.4% (OptLayerPolicy) compared to 86.1% (OptLayer) and that the policy after training is increased to 104.9% (GreyOptLayerPolicy) compared to 103.4% (OptLayer) - all relative to a vanilla RL benchmark. Although introducing surrogate functions into the optimisation problem requires special attention, we conclude that the newly presented GreyOptLayerPolicy method is the most advantageous.