# Action Set Based Policy Optimization for Safe Power Grid Management

arXiv: https://arxiv.org/abs/2106.15200 · HTML: https://mlsanity.com/paper/2106.15200/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Learning to run a power network challenge for training topology controllers](https://mlsanity.com/paper/1912.04211/) (2019)
- [Adaptive Power System Emergency Control using Deep Reinforcement Learning](https://mlsanity.com/paper/1903.03712/) (2019)
- [Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.09374/) (2019)
- [Solving Online Threat Screening Games using Constrained Action Space Reinforcement Learning](https://mlsanity.com/paper/1911.08799/) (2019)
- [Autonomous Voltage Control for Grid Operation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.10597/) (2019)
- [Hierarchical Decision Making In Electricity Grid Management](https://mlsanity.com/paper/1603.01840/) (2016)

## Built on by (most cited first, 21 of 21)

- [Power Grid Congestion Management via Topology Optimization with AlphaZero](https://mlsanity.com/paper/2211.05612/) (2022)
- [PowRL: A Reinforcement Learning Framework for Robust Management of Power Networks](https://mlsanity.com/paper/2212.02397/) (2022)
- [Learning to run a power network with trust](https://mlsanity.com/paper/2110.12908/) (2021)
- [Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents](https://mlsanity.com/paper/2304.00765/) (2023)
- [Hierarchical Reinforcement Learning for Power Network Topology Control](https://mlsanity.com/paper/2311.02129/) (2023)
- [HUGO -- Highlighting Unseen Grid Options: Combining Deep Reinforcement Learning with a Heuristic Target Topology Approach](https://mlsanity.com/paper/2405.00629/) (2024)
- [Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training](https://mlsanity.com/paper/2110.08956/) (2021)
- [Reinforcement learning for Energies of the future and carbon neutrality: a Challenge Design](https://mlsanity.com/paper/2207.10330/) (2022)
- [Imitation Learning for Intra-Day Power Grid Operation through Topology Actions](https://mlsanity.com/paper/2407.19865/) (2024)
- [Fault Detection for agents on power grid topology optimization: A Comprehensive analysis](https://mlsanity.com/paper/2406.16426/) (2024)
- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [Graph Neural Networks for Transmission Grid Topology Control: Busbar Information Asymmetry and Heterogeneous Representations](https://mlsanity.com/paper/2501.07186/) (2025)
- [Learning Topology Actions for Power Grid Control: A Graph-Based Soft-Label Imitation Learning Approach](https://mlsanity.com/paper/2503.15190/) (2025)
- [Graph-Enhanced Model-Free Reinforcement Learning Agents for Efficient Power Grid Topological Control](https://mlsanity.com/paper/2503.20688/) (2025)
- [A Critical Review of Safe Reinforcement Learning Techniques in Smart Grid Applications](https://mlsanity.com/paper/2409.16256/) (2024)
- [Physics Informed Reinforcement Learning with Gibbs Priors for Topology Control in Power Grids](https://mlsanity.com/paper/2604.01830/) (2026)
- [Multi-Agent Reinforcement Learning for Power Grid Topology Optimization](https://mlsanity.com/paper/2310.02605/) (2023)
- [Interpretable Policy Distillation for Power Grid Topology Control](https://mlsanity.com/paper/2606.00561/) (2026)
- [Reinforcement Learning versus Optimization for Optimal Transmission Switching: A Comparative Study](https://mlsanity.com/paper/2607.10948/) (2026)
- [A Comparative Study of Graph Representations for GNN-Based Power Grid Control in L2RPN](https://mlsanity.com/paper/2609.02538/) (2026)
- [Data-Assimilation-Assisted Reinforcement Learning for Power Grid Control under Load Uncertainty](https://mlsanity.com/paper/2609.27229/) (2026)

## Abstract (arXiv)

Maintaining the stability of the modern power grid is becoming increasingly difficult due to fluctuating power consumption, unstable power supply coming from renewable energies, and unpredictable accidents such as man-made and natural disasters. As the operation on the power grid must consider its impact on future stability, reinforcement learning (RL) has been employed to provide sequential decision-making in power grid management. However, existing methods have not considered the environmental constraints. As a result, the learned policy has risk of selecting actions that violate the constraints in emergencies, which will escalate the issue of overloaded power lines and lead to large-scale blackouts. In this work, we propose a novel method for this problem, which builds on top of the search-based planning algorithm. At the planning stage, the search space is limited to the action set produced by the policy. The selected action strictly follows the constraints by testing its outcome with the simulation function provided by the system. At the learning stage, to address the problem that gradients cannot be propagated to the policy, we introduce Evolutionary Strategies (ES) with black-box policy optimization to improve the policy directly, maximizing the returns of the long run. In NeurIPS 2020 Learning to Run Power Network (L2RPN) competition, our solution safely managed the power grid and ranked first in both tracks.