# Reducing Action Space: Reference-Model-Assisted Deep Reinforcement Learning for Inverter-based Volt-Var Control

arXiv: https://arxiv.org/abs/2210.07360 · HTML: https://mlsanity.com/paper/2210.07360/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [pandapower - an Open Source Python Tool for Convenient Modeling, Analysis and Optimization of Electric Power Systems](https://mlsanity.com/paper/1709.06743/) (2017)
- [Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges](https://mlsanity.com/paper/2102.01168/) (2021)
- [Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning](https://mlsanity.com/paper/1904.09374/) (2019)
- [Consensus Multi-Agent Reinforcement Learning for Volt-VAR Control in Power Distribution Networks](https://mlsanity.com/paper/2007.02991/) (2020)
- [Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems](https://mlsanity.com/paper/2003.03681/) (2020)
- [Online Multi-agent Reinforcement Learning for Decentralized Inverter-based Volt-VAR Control](https://mlsanity.com/paper/2006.12841/) (2020)
- [Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks](https://mlsanity.com/paper/2005.11142/) (2020)
- [Stochastic Reactive Power Management in Microgrids with Renewables](https://mlsanity.com/paper/1409.6758/) (2014)
- [Photovoltaic Inverter Controllers Seeking AC Optimal Power Flow Solutions](https://mlsanity.com/paper/1501.00188/) (2014)

## Abstract (arXiv)

Reference-model-assisted deep reinforcement learning (DRL) for inverter-based Volt-Var Control (IB-VVC) in active distribution networks is proposed. We investigate that a large action space increases the learning difficulties of DRL and degrades the optimization performance in the process of generating data and training neural networks. To reduce the action space of DRL, we design a reference-model-assisted DRL approach. We introduce definitions of the reference model, reference-model-based optimization, and reference actions. The reference-model-assisted DRL learns the residual actions between the reference actions and optimal actions, rather than learning the optimal actions directly. Since the residual actions are considerably smaller than the optimal actions for a reference model, we can design a smaller action space for the reference-model-assisted DRL. It reduces the learning difficulties of DRL and optimises the performance of the reference-model-assisted DRL approach. It is noteworthy that the reference-model-assisted DRL approach is compatible with any policy gradient DRL algorithms for continuous action problems. This work takes the soft actor-critic algorithm as an example and designs a reference-model-assisted soft actor-critic algorithm. Simulations show that 1) large action space degrades the performance of DRL in the whole training stage, and 2) reference-model-assisted DRL requires fewer iteration times and returns a better optimization performance.