# Reducing Learning Difficulties: One-Step Two-Critic Deep Reinforcement Learning for Inverter-based Volt-Var Control

arXiv: https://arxiv.org/abs/2203.16289 · HTML: https://mlsanity.com/paper/2203.16289/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Better Exploration with Optimistic Actor-Critic](https://mlsanity.com/paper/1910.12807/) (2019)
- [pandapower - an Open Source Python Tool for Convenient Modeling, Analysis and Optimization of Electric Power Systems](https://mlsanity.com/paper/1709.06743/) (2017)
- [Softmax Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2010.09177/) (2020)
- [Efficient Contextual Bandits with Continuous Actions](https://mlsanity.com/paper/2006.06040/) (2020)
- [Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.09374/) (2019)
- [Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks](https://mlsanity.com/paper/2007.02991/) (2020)
- [Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems](https://mlsanity.com/paper/2003.03681/) (2020)
- [Online Multi-agent Reinforcement Learning for Decentralized Inverter-based Volt-VAR Control](https://mlsanity.com/paper/2006.12841/) (2020)
- [Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks](https://mlsanity.com/paper/2005.11142/) (2020)
- [Stochastic Reactive Power Management in Microgrids with Renewables](https://mlsanity.com/paper/1409.6758/) (2014)
- [Bi-level Off-policy Reinforcement Learning for Volt/VAR Control Involving Continuous and Discrete Devices](https://mlsanity.com/paper/2104.05902/) (2021)

## Built on by (most cited first, 1 of 1)

- [Residual Deep Reinforcement Learning for Inverter-based Volt-Var Control](https://mlsanity.com/paper/2408.06790/) (2024)

## Abstract (arXiv)

A one-step two-critic deep reinforcement learning (OSTC-DRL) approach for inverter-based volt-var control (IB-VVC) in active distribution networks is proposed in this paper. Firstly, considering IB-VVC can be formulated as a single-period optimization problem, we formulate the IB-VVC as a one-step Markov decision process rather than the standard Markov decision process, which simplifies the DRL learning task. Then we design the one-step actor-critic DRL scheme which is a simplified version of recent DRL algorithms, and it avoids the issue of Q value overestimation successfully. Furthermore, considering two objectives of VVC: minimizing power loss and eliminating voltage violation, we utilize two critics to approximate the rewards of two objectives separately. It simplifies the approximation tasks of each critic, and avoids the interaction effect between two objectives in the learning process of critic. The OSTC-DRL approach integrates the one-step actor-critic DRL scheme and the two-critic technology. Based on the OSTC-DRL, we design two centralized DRL algorithms. Further, we extend the OSTC-DRL to multi-agent OSTC-DRL for decentralized IB-VVC and design two multi-agent DRL algorithms. Simulations demonstrate that the proposed OSTC-DRL has a faster convergence rate and a better control performance, and the multi-agent OSTC-DRL works well for decentralized IB-VVC problems.