# Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility

arXiv: https://arxiv.org/abs/2203.03417 · HTML: https://mlsanity.com/paper/2203.03417/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2022.

## Built on by (most cited first, 11 of 11)

- [Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey](https://mlsanity.com/paper/2504.21048/) (2025)
- [The Synergy Between Optimal Transport Theory and Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.10949/) (2024)
- [Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility](https://mlsanity.com/paper/2305.18875/) (2023)
- [Home Electricity Data Generator (HEDGE): An open-access tool for the generation of electric vehicle, residential demand, and PV generation profiles](https://mlsanity.com/paper/2310.01661/) (2023)
- [Reinforcement Learning Enabled Peer-to-Peer Energy Trading for Dairy Farms](https://mlsanity.com/paper/2405.12716/) (2024)
- [Multi-Agent Reinforcement Learning for Energy Networks: Computational Challenges, Progress and Open Problems](https://mlsanity.com/paper/2404.15583/) (2024)
- [GradMAP: Gradient-Based Multi-Agent Proximal Learning for Grid-Edge Flexibility](https://mlsanity.com/paper/2604.24549/) (2026)
- [Peer-to-Peer Energy Trading in Dairy Farms using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.23148/) (2025)
- [Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Markets](https://mlsanity.com/paper/2602.16062/) (2026)
- [Hybrid Human-Agent Social Dilemmas in Energy Markets](https://mlsanity.com/paper/2603.11834/) (2026)
- [Supervised Reinforcement Learning for the Coordination of Distributed Energy Resources](https://mlsanity.com/paper/2606.24947/) (2026)

## Abstract (arXiv)

This paper proposes a novel scalable type of multi-agent reinforcement learning-based coordination for distributed residential energy. Cooperating agents learn to control the flexibility offered by electric vehicles, space heating and flexible loads in a partially observable stochastic environment. In the standard independent Q-learning approach, the coordination performance of agents under partial observability drops at scale in stochastic environments. Here, the novel combination of learning from off-line convex optimisations on historical data and isolating marginal contributions to total rewards in reward signals increases stability and performance at scale. Using fixed-size Q-tables, prosumers are able to assess their marginal impact on total system objectives without sharing personal data either with each other or with a central coordinator. Case studies are used to assess the fitness of different combinations of exploration sources, reward definitions, and multi-agent learning frameworks. It is demonstrated that the proposed strategies create value at individual and system levels thanks to reductions in the costs of energy imports, losses, distribution network congestion, battery depreciation and greenhouse gas emissions.