# Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1904.09374 · HTML: https://mlsanity.com/paper/1904.09374/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges](https://mlsanity.com/paper/2102.01168/) (2021)
- [Action Set Based Policy Optimization for Safe Power Grid Management](https://mlsanity.com/paper/2106.15200/) (2021)
- [PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems](https://mlsanity.com/paper/2111.05969/) (2021)
- [Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications](https://mlsanity.com/paper/2401.09339/) (2024)
- [Transmission Interface Power Flow Adjustment: A Deep Reinforcement Learning Approach based on Multi-task Attribution Map](https://mlsanity.com/paper/2405.15831/) (2024)
- [Risk-Aware Learning for Scalable Voltage Optimization in Distribution Grids](https://mlsanity.com/paper/2110.01490/) (2021)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [Powerformer: A Section-adaptive Transformer for Power Flow Adjustment](https://mlsanity.com/paper/2401.02771/) (2024)
- [Bi-level Off-policy Reinforcement Learning for Volt/VAR Control Involving Continuous and Discrete Devices](https://mlsanity.com/paper/2104.05902/) (2021)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [Transfer Deep Reinforcement Learning-based Large-scale V2G Continuous Charging Coordination with Renewable Energy Sources](https://mlsanity.com/paper/2210.07013/) (2022)
- [Reducing Learning Difficulties: One-Step Two-Critic Deep Reinforcement Learning for Inverter-based Volt-Var Control](https://mlsanity.com/paper/2203.16289/) (2022)
- [Deep Reinforcement Learning for Voltage Control and Renewable Accommodation Using Spatial-Temporal Graph Information](https://mlsanity.com/paper/2401.15848/) (2024)
- [Residual Deep Reinforcement Learning for Inverter-based Volt-Var Control](https://mlsanity.com/paper/2408.06790/) (2024)
- [Reinforcement Learning to Initialize Newton-Raphson for AC Power Flow with Quantum Annealing-Based Environment Updates](https://mlsanity.com/paper/2511.20237/) (2025)
- [Using Deep Reinforcement Learning to solve Optimal Power Flow problem with generator failures](https://mlsanity.com/paper/2205.02108/) (2022)
- [Reducing Action Space: Reference-Model-Assisted Deep Reinforcement Learning for Inverter-based Volt-Var Control](https://mlsanity.com/paper/2210.07360/) (2022)
- [Physics-Informed Reinforcement Learning for Large-Scale EV Smart Charging Considering Distribution Network Voltage Constraints](https://mlsanity.com/paper/2510.12335/) (2025)
- [Reinforcement Learning for Vehicle-to-Grid Voltage Regulation: Single-Hub to Multi-Hub Coordination with Battery-Aware Constraints](https://mlsanity.com/paper/2603.07237/) (2026)
- [A Human-AI Teaming Framework for Deep Reinforcement Learning-Based Voltage Regulation in Distribution Networks](https://mlsanity.com/paper/2607.19667/) (2026)

## Abstract (arXiv)

Modern distribution grids are currently being challenged by frequent and sizable voltage fluctuations, due mainly to the increasing deployment of electric vehicles and renewable generators. Existing approaches to maintaining bus voltage magnitudes within the desired region can cope with either traditional utility-owned devices (e.g., shunt capacitors), or contemporary smart inverters that come with distributed generation units (e.g., photovoltaic plants). The discrete on-off commitment of capacitor units is often configured on an hourly or daily basis, yet smart inverters can be controlled within milliseconds, thus challenging joint control of these two types of assets. In this context, a novel two-timescale voltage regulation scheme is developed for distribution grids by judiciously coupling data-driven with physicsbased optimization. On a faster timescale, say every second, the optimal setpoints of smart inverters are obtained by minimizing instantaneous bus voltage deviations from their nominal values, based on either the exact alternating current power flow model or a linear approximant of it; whereas, on the slower timescale (e.g., every hour), shunt capacitors are configured to minimize the longterm discounted voltage deviations using a deep reinforcement learning algorithm. Extensive numerical tests on a real-world 47- bus distribution network as well as the IEEE 123-bus test feeder using real data corroborate the effectiveness of the novel scheme.