# Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges

arXiv: https://arxiv.org/abs/2102.01168 · HTML: https://mlsanity.com/paper/2102.01168/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 205 references that are in the map.
- First posted on arXiv 27 Jan 2021.

## Builds on (most cited first, 30 of 64)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift](https://mlsanity.com/paper/1908.00261/) (2019)
- [Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning](https://mlsanity.com/paper/1902.00923/) (2019)
- [Robust Deep Reinforcement Learning with Adversarial Attacks](https://mlsanity.com/paper/1712.03632/) (2017)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learning](https://mlsanity.com/paper/2002.00260/) (2020)
- [Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction](https://mlsanity.com/paper/2006.03041/) (2020)

## Built on by (most cited first, 30 of 50)

- [Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.17116/) (2022)
- [Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning](https://mlsanity.com/paper/2502.19652/) (2025)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [PDE Control Gym: A Benchmark for Data-Driven Boundary Control of Partial Differential Equations](https://mlsanity.com/paper/2405.11401/) (2024)
- [Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL](https://mlsanity.com/paper/2405.18520/) (2024)
- [X-GridAgent: An LLM-Powered Agentic AI System for Assisting Power Grid Analysis](https://mlsanity.com/paper/2512.20789/) (2025)
- [Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation](https://mlsanity.com/paper/2405.20860/) (2024)
- [RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations](https://mlsanity.com/paper/2503.23101/) (2025)
- [Contingency-constrained economic dispatch with safe reinforcement learning](https://mlsanity.com/paper/2205.06212/) (2022)
- [Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidance](https://mlsanity.com/paper/2212.01939/) (2022)
- [Contextual Restless Multi-Armed Bandits with Application to Demand Response Decision-Making](https://mlsanity.com/paper/2403.15640/) (2024)
- [Transmission Interface Power Flow Adjustment: A Deep Reinforcement Learning Approach based on Multi-task Attribution Map](https://mlsanity.com/paper/2405.15831/) (2024)
- [Federated Reinforcement Learning for Real-Time Electric Vehicle Charging and Discharging Control](https://mlsanity.com/paper/2210.01452/) (2022)
- [CommonPower: A Framework for Safe Data-Driven Smart Grid Control](https://mlsanity.com/paper/2406.03231/) (2024)
- [A Survey of Reinforcement Learning for Optimization in Automation](https://mlsanity.com/paper/2502.09417/) (2025)
- [A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions](https://mlsanity.com/paper/2508.09128/) (2025)
- [Safe Reinforcement Learning for Power System Control: A Review](https://mlsanity.com/paper/2407.00681/) (2024)
- [Online Event-Triggered Switching for Frequency Control in Power Grids with Variable Inertia](https://mlsanity.com/paper/2408.15436/) (2024)
- [Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach](https://mlsanity.com/paper/2510.18687/) (2025)
- [Data-Driven Linear Koopman Embedding for Networked Systems: Model-Predictive Grid Control](https://mlsanity.com/paper/2206.01272/) (2022)
- [Control invariant set enhanced safe reinforcement learning: improved sampling efficiency, guaranteed stability and robustness](https://mlsanity.com/paper/2305.15602/) (2023)
- [Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control](https://mlsanity.com/paper/2405.08443/) (2024)
- [PseudoAct: Leveraging Pseudocode Synthesis for Flexible Planning and Action Control in Large Language Model Agents](https://mlsanity.com/paper/2602.23668/) (2026)
- [Control invariant set enhanced reinforcement learning for process control: improved sampling efficiency and guaranteed stability](https://mlsanity.com/paper/2304.05509/) (2023)
- [A Multiobjective Reinforcement Learning Framework for Microgrid Energy Management](https://mlsanity.com/paper/2307.08692/) (2023)
- [Out-of-Distribution-Aware Electric Vehicle Charging](https://mlsanity.com/paper/2311.05941/) (2023)
- [Distributed Policy Gradient for Linear Quadratic Networked Control with Limited Communication Range](https://mlsanity.com/paper/2403.03055/) (2024)
- [Imitation Game: A Model-based and Imitation Learning Deep Reinforcement Learning Hybrid](https://mlsanity.com/paper/2404.01794/) (2024)
- [Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks](https://mlsanity.com/paper/2408.06776/) (2024)
- [Residual Deep Reinforcement Learning for Inverter-based Volt-Var Control](https://mlsanity.com/paper/2408.06790/) (2024)

## Abstract (arXiv)

With large-scale integration of renewable generation and distributed energy resources, modern power systems are confronted with new operational challenges, such as growing complexity, increasing uncertainty, and aggravating volatility. Meanwhile, more and more data are becoming available owing to the widespread deployment of smart meters, smart sensors, and upgraded communication networks. As a result, data-driven control techniques, especially reinforcement learning (RL), have attracted surging attention in recent years. This paper provides a comprehensive review of various RL techniques and how they can be applied to decision-making and control in power systems. In particular, we select three key applications, i.e., frequency regulation, voltage control, and energy management, as examples to illustrate RL-based models and solutions. We then present the critical issues in the application of RL, i.e., safety, robustness, scalability, and data. Several potential future directions are discussed as well.