# Real-world challenges for multi-agent reinforcement learning in grid-interactive buildings

arXiv: https://arxiv.org/abs/2112.06127 · HTML: https://mlsanity.com/paper/2112.06127/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 52 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 10 of 10)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.00714/) (2021)
- [Physics-constrained Deep Learning of Multi-zone Building Thermal Dynamics](https://mlsanity.com/paper/2011.05987/) (2020)
- [CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management](https://mlsanity.com/paper/2012.10504/) (2020)
- [A Centralised Soft Actor Critic Deep Reinforcement Learning Approach to District Demand Side Management through CityLearn](https://mlsanity.com/paper/2009.10562/) (2020)

## Built on by (most cited first, 9 of 9)

- [CityLearn v2: Energy-flexible, resilient, occupant-centric, and carbon-aware management of grid-interactive communities](https://mlsanity.com/paper/2405.03848/) (2024)
- [Safe reinforcement learning for multi-energy management systems with known constraint functions](https://mlsanity.com/paper/2207.03830/) (2022)
- [MERLIN: Multi-agent offline and transfer learning for occupant-centric energy flexible operation of grid-interactive communities using smart meter data and CityLearn](https://mlsanity.com/paper/2301.01148/) (2022)
- [Learning to Do or Learning While Doing: Reinforcement Learning and Bayesian Optimisation for Online Continuous Tuning](https://mlsanity.com/paper/2306.03739/) (2023)
- [Combined Peak Reduction and Self-Consumption Using Proximal Policy Optimization](https://mlsanity.com/paper/2211.14831/) (2022)
- [Control of Renewable Energy Communities using AI and Real-World Data](https://mlsanity.com/paper/2505.17321/) (2025)
- [Real-world validation of safe reinforcement learning, model predictive control and decision tree-based home energy management systems](https://mlsanity.com/paper/2408.07435/) (2024)
- [Generalising Battery Control in Net-Zero Buildings via Personalised Federated RL](https://mlsanity.com/paper/2412.20946/) (2024)
- [ADAPT: Physics-Aware Diffusion-based World Models for Adaptive Predictive Transferable HVAC Control](https://mlsanity.com/paper/2608.19804/) (2026)

## Abstract (arXiv)

Building upon prior research that highlighted the need for standardizing environments for building control research, and inspired by recently introduced challenges for real life reinforcement learning control, here we propose a non-exhaustive set of nine real world challenges for reinforcement learning control in grid-interactive buildings. We argue that research in this area should be expressed in this framework in addition to providing a standardized environment for repeatability. Advanced controllers such as model predictive control and reinforcement learning (RL) control have both advantages and disadvantages that prevent them from being implemented in real world problems. Comparisons between the two are rare, and often biased. By focusing on the challenges, we can investigate the performance of the controllers under a variety of situations and generate a fair comparison. As a demonstration, we implement the offline learning challenge in CityLearn and study the impact of different levels of domain knowledge and complexity of RL algorithms. We show that the sequence of operations utilized in a rule based controller (RBC) used for offline training affects the performance of the RL agents when evaluated on a set of four energy flexibility metrics. Longer offline learning from an optimized RBC leads to improved performance in the long run. RL agents that learn from a simplified RBC risk poorer performance as the offline learning period increases. We also observe no impact on performance from information sharing amongst agents. We call for a more interdisciplinary effort of the research community to address the real world challenges, and unlock the potential of grid-interactive building