# Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility

arXiv: https://arxiv.org/abs/2305.18875 · HTML: https://mlsanity.com/paper/2305.18875/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 87 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.10800/) (2020)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks](https://mlsanity.com/paper/2110.14300/) (2021)
- [Autoregressive Convolutional Neural Networks for Asynchronous Time Series](https://mlsanity.com/paper/1703.04122/) (2017)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [GridLearn: Multiagent Reinforcement Learning for Grid-Aware Building Energy Management](https://mlsanity.com/paper/2110.06396/) (2021)
- [Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility](https://mlsanity.com/paper/2203.03417/) (2022)
- [A Learning-boosted Quasi-Newton Method for AC Optimal Power Flow](https://mlsanity.com/paper/2007.06074/) (2020)
- [QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.04197/) (2020)
- [Coordination of resources at the edge of the electricity grid: systematic review and taxonomy](https://mlsanity.com/paper/2202.03786/) (2022)

## Built on by (most cited first, 5 of 5)

- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [GradMAP: Gradient-Based Multi-Agent Proximal Learning for Grid-Edge Flexibility](https://mlsanity.com/paper/2604.24549/) (2026)
- [Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Markets](https://mlsanity.com/paper/2602.16062/) (2026)
- [MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets](https://mlsanity.com/paper/2602.16063/) (2026)
- [Finite-Sample Probabilistic Safety Certification for AI-Based Grid-Edge Coordination](https://mlsanity.com/paper/2609.28182/) (2026)

## Abstract (arXiv)

This paper investigates how deep multi-agent reinforcement learning can enable the scalable and privacy-preserving coordination of residential energy flexibility. The coordination of distributed resources such as electric vehicles and heating will be critical to the successful integration of large shares of renewable energy in our electricity grid and, thus, to help mitigate climate change. The pre-learning of individual reinforcement learning policies can enable distributed control with no sharing of personal data required during execution. However, previous approaches for multi-agent reinforcement learning-based distributed energy resources coordination impose an ever greater training computational burden as the size of the system increases. We therefore adopt a deep multi-agent actor-critic method which uses a centralised but factored critic to rehearse coordination ahead of execution. Results show that coordination is achieved at scale, with minimal information and communication infrastructure requirements, no interference with daily activities, and privacy protection. Significant savings are obtained for energy users, the distribution network and greenhouse gas emissions. Moreover, training times are nearly 40 times shorter than with a previous state-of-the-art reinforcement learning approach without the factored critic for 30 homes.