# Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training

arXiv: https://arxiv.org/abs/2110.08956 · HTML: https://mlsanity.com/paper/2110.08956/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles](https://mlsanity.com/paper/1906.03973/) (2019)
- [Accelerated Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1803.02811/) (2018)
- [Action Set Based Policy Optimization for Safe Power Grid Management](https://mlsanity.com/paper/2106.15200/) (2021)
- [Reinforcement Learning for Electricity Network Operation](https://mlsanity.com/paper/2003.07339/) (2020)
- [Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks](https://mlsanity.com/paper/2005.11142/) (2020)
- [Adversarial Training for a Continuous Robustness Control Problem in Power Systems](https://mlsanity.com/paper/2012.11390/) (2020)
- [Understanding the Safety Requirements for Learning-based Power Systems Operations](https://mlsanity.com/paper/2110.04983/) (2021)
- [Estimating Distribution Grid Topologies: A Graphical Learning based Approach](https://mlsanity.com/paper/1602.08509/) (2016)
- [Chance-Constrained AC Optimal Power Flow: Reformulations and Efficient Algorithms](https://mlsanity.com/paper/1706.03241/) (2017)

## Built on by (most cited first, 8 of 8)

- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [SoK: Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2301.04299/) (2023)
- [Imitation Learning for Intra-Day Power Grid Operation through Topology Actions](https://mlsanity.com/paper/2407.19865/) (2024)
- [Semi-Supervised Multi-Task Learning Based Framework for Power System Security Assessment](https://mlsanity.com/paper/2407.08886/) (2024)
- [Data Poisoning: An Overlooked Threat to Power Grid Resilience](https://mlsanity.com/paper/2407.14684/) (2024)
- [Bayesian Reinforcement Learning for Automatic Voltage Control under Cyber-Induced Uncertainty](https://mlsanity.com/paper/2305.16469/) (2023)
- [A Reinforcement Learning Engine with Reduced Action and State Space for Scalable Cyber-Physical Optimal Response](https://mlsanity.com/paper/2410.04518/) (2024)
- [Data-Assimilation-Assisted Reinforcement Learning for Power Grid Control under Load Uncertainty](https://mlsanity.com/paper/2609.27229/) (2026)

## Abstract (arXiv)

Due to the proliferation of renewable energy and its intrinsic intermittency and stochasticity, current power systems face severe operational challenges. Data-driven decision-making algorithms from reinforcement learning (RL) offer a solution towards efficiently operating a clean energy system. Although RL algorithms achieve promising performance compared to model-based control models, there has been limited investigation of RL robustness in safety-critical physical systems. In this work, we first show that several competition-winning, state-of-the-art RL agents proposed for power system control are vulnerable to adversarial attacks. Specifically, we use an adversary Markov Decision Process to learn an attack policy, and demonstrate the potency of our attack by successfully attacking multiple winning agents from the Learning To Run a Power Network (L2RPN) challenge, under both white-box and black-box attack settings. We then propose to use adversarial training to increase the robustness of RL agent against attacks and avoid infeasible operational decisions. To the best of our knowledge, our work is the first to highlight the fragility of grid control RL algorithms, and contribute an effective defense scheme towards improving their robustness and security.