# Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks

arXiv: https://arxiv.org/abs/2110.14300 · HTML: https://mlsanity.com/paper/2110.14300/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 60 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [pandapower - an Open Source Python Tool for Convenient Modeling, Analysis and Optimization of Electric Power Systems](https://mlsanity.com/paper/1709.06743/) (2017)
- [Shapley Q-value: A Local Reward Approach to Solve Global Reward Games](https://mlsanity.com/paper/1907.05707/) (2019)
- [Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics](https://mlsanity.com/paper/1910.01465/) (2019)
- [Efficient Contextual Bandits with Continuous Actions](https://mlsanity.com/paper/2006.06040/) (2020)
- [Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems](https://mlsanity.com/paper/2103.07932/) (2021)
- [Online Multi-agent Reinforcement Learning for Decentralized Inverter-based Volt-VAR Control](https://mlsanity.com/paper/2006.12841/) (2020)
- [Real-Time Local Volt/VAR Control Under External Disturbances with High PV Penetration](https://mlsanity.com/paper/1710.02551/) (2017)
- [A Fully Distributed Reactive Power Optimization and Control Method for Active Distribution Networks](https://mlsanity.com/paper/1407.5032/) (2014)
- [Distributed Voltage Regulation of Active Distribution System Based on Enhanced Multi-agent Deep Reinforcement Learning](https://mlsanity.com/paper/2006.00546/) (2020)
- [Design and implementation of an environment for Learning to Run a Power Network (L2RPN)](https://mlsanity.com/paper/2104.04080/) (2021)

## Built on by (most cited first, 30 of 54)

- [Multi-agent Dynamic Algorithm Configuration](https://mlsanity.com/paper/2210.06835/) (2022)
- [Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2204.07932/) (2022)
- [Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition](https://mlsanity.com/paper/2211.12712/) (2022)
- [Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?](https://mlsanity.com/paper/2305.17352/) (2023)
- [Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey](https://mlsanity.com/paper/2504.21048/) (2025)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [RL2: Reinforce Large Language Model to Assist Safe Reinforcement Learning for Energy Management of Active Distribution Networks](https://mlsanity.com/paper/2412.01303/) (2024)
- [Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2207.03902/) (2022)
- [Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.00521/) (2023)
- [Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization](https://mlsanity.com/paper/2207.02016/) (2022)
- [Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generation](https://mlsanity.com/paper/2305.05116/) (2023)
- [Open Ad Hoc Teamwork with Cooperative Game Theory](https://mlsanity.com/paper/2402.15259/) (2024)
- [Stabilizing Voltage in Power Distribution Networks via Multi-Agent Reinforcement Learning with Transformer](https://mlsanity.com/paper/2206.03721/) (2022)
- [MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.08244/) (2022)
- [Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility](https://mlsanity.com/paper/2305.18875/) (2023)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2401.04934/) (2024)
- [Best Possible Q-Learning](https://mlsanity.com/paper/2302.01188/) (2023)
- [Networked Multiagent Safe Reinforcement Learning for Low-carbon Demand Management in Distribution Network](https://mlsanity.com/paper/2311.15594/) (2023)
- [RL-ADN: A High-Performance Deep Reinforcement Learning Environment for Optimal Energy Storage Systems Dispatch in Active Distribution Networks](https://mlsanity.com/paper/2408.03685/) (2024)
- [Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2506.11285/) (2025)
- [Robust Multi-agent Communication via Multi-view Message Certification](https://mlsanity.com/paper/2305.13936/) (2023)
- [Approximate Global Convergence of Independent Learning in Multi-Agent Systems](https://mlsanity.com/paper/2405.19811/) (2024)
- [A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.17697/) (2025)
- [Influence-Augmented Local Simulators: A Scalable Solution for Fast Deep RL in Large Networked Systems](https://mlsanity.com/paper/2202.01534/) (2022)
- [Multi-Agent Reinforcement Learning for Fast-Timescale Demand Response of Residential Loads](https://mlsanity.com/paper/2301.02593/) (2023)
- [Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2302.09605/) (2023)
- [A Constraint Enforcement Deep Reinforcement Learning Framework for Optimal Energy Storage Systems Dispatch](https://mlsanity.com/paper/2307.14304/) (2023)
- [Federated Reinforcement Learning for Electric Vehicles Charging Control on Distribution Networks](https://mlsanity.com/paper/2308.08792/) (2023)
- [Multi-Agent Reinforcement Learning for Energy Networks: Computational Challenges, Progress and Open Problems](https://mlsanity.com/paper/2404.15583/) (2024)
- [Temporal Prototype-Aware Learning for Active Voltage Control on Power Distribution Networks](https://mlsanity.com/paper/2406.17818/) (2024)

## Abstract (arXiv)

This paper presents a problem in power networks that creates an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL). The emerging trend of decarbonisation is placing excessive stress on power distribution networks. Active voltage control is seen as a promising solution to relieve power congestion and improve voltage quality without extra hardware investment, taking advantage of the controllable apparatuses in the network, such as roof-top photovoltaics (PVs) and static var compensators (SVCs). These controllable apparatuses appear in a vast number and are distributed in a wide geographic area, making MARL a natural candidate. This paper formulates the active voltage control problem in the framework of Dec-POMDP and establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community and be a drive force towards real-world applications of MARL algorithms. Finally, we analyse the special characteristics of the active voltage control problems that cause challenges (e.g. interpretability) for state-of-the-art MARL approaches, and summarise the potential directions.