# RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operations

arXiv: https://arxiv.org/abs/2503.23101 · HTML: https://mlsanity.com/paper/2503.23101/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 29 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Learning to run a Power Network Challenge: a Retrospective Analysis](https://mlsanity.com/paper/2103.03104/) (2021)
- [Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges](https://mlsanity.com/paper/2102.01168/) (2021)
- [Learning to run a power network challenge for training topology controllers](https://mlsanity.com/paper/1912.04211/) (2019)
- [CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management](https://mlsanity.com/paper/2012.10504/) (2020)
- [Reinforcement Learning for Electricity Network Operation](https://mlsanity.com/paper/2003.07339/) (2020)
- [PowRL: A Reinforcement Learning Framework for Robust Management of Power Networks](https://mlsanity.com/paper/2212.02397/) (2022)
- [Learning to run a power network with trust](https://mlsanity.com/paper/2110.12908/) (2021)
- [Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents](https://mlsanity.com/paper/2304.00765/) (2023)
- [A Review of Safe Reinforcement Learning Methods for Modern Power Systems](https://mlsanity.com/paper/2407.00304/) (2024)
- [pymgrid: An Open-Source Python Microgrid Simulator for Applied Artificial Intelligence Research](https://mlsanity.com/paper/2011.08004/) (2020)
- [Safe Deep Reinforcement Learning by Verifying Task-Level Properties](https://mlsanity.com/paper/2302.10030/) (2023)
- [Curriculum Based Reinforcement Learning of Grid Topology Controllers to Prevent Thermal Cascading](https://mlsanity.com/paper/2112.09996/) (2021)
- [Improving Deep Policy Gradients with Value Function Search](https://mlsanity.com/paper/2302.10145/) (2023)
- [Gym-ANM: Open-source software to leverage reinforcement learning for power system management in research and education](https://mlsanity.com/paper/2105.08846/) (2021)
- [Improving Policy Optimization via varepsilon-Retrain](https://mlsanity.com/paper/2406.08315/) (2024)
- [On Stateful Value Factorization in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.15381/) (2024)
- [Real-time scheduling of renewable power systems through planning-based reinforcement learning](https://mlsanity.com/paper/2303.05205/) (2023)

## Built on by (most cited first, 9 of 9)

- [Audited Selective Verification for Risk-Controlled N-1 Thermal Contingency Screening under Deployment Shift](https://mlsanity.com/paper/2607.13221/) (2026)
- [OpenG2G: A Simulation Platform for AI Datacenter-Grid Runtime Coordination](https://mlsanity.com/paper/2605.05519/) (2026)
- [Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.14758/) (2026)
- [Interpretable Policy Distillation for Power Grid Topology Control](https://mlsanity.com/paper/2606.00561/) (2026)
- [Robustness of Reinforcement Learning-Based Congestion Management in Low-Voltage Grids](https://mlsanity.com/paper/2607.16004/) (2026)
- [A Survey on the Verification of Reinforcement Learning Policies](https://mlsanity.com/paper/2607.16210/) (2026)
- [Graph Machine Learning: An Opportunity for Power Systems](https://mlsanity.com/paper/2608.16494/) (2026)
- [PowerZooJax: A JAX-based Power System Benchmark for Reinforcement Learning](https://mlsanity.com/paper/2609.36052/) (2026)
- [Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery](https://mlsanity.com/paper/2610.03872/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) can provide adaptive and scalable controllers essential for power grid decarbonization. However, RL methods struggle with power grids' complex dynamics, long-horizon goals, and hard physical constraints. For these reasons, we present RL2Grid, a benchmark designed in collaboration with power system operators to accelerate progress in grid control and foster RL maturity. Built on RTE France's power simulation framework, RL2Grid standardizes tasks, state and action spaces, and reward structures for a systematic evaluation and comparison of RL algorithms. Moreover, we integrate operational heuristics and design safety constraints based on human expertise to ensure alignment with physical requirements. By establishing reference performance metrics for classic RL baselines on RL2Grid's tasks, we highlight the need for novel methods capable of handling real systems and discuss future directions for RL-based grid control.