# Trainability issues in quantum policy gradients

arXiv: https://arxiv.org/abs/2406.09614 · HTML: https://mlsanity.com/paper/2406.09614/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Quantum reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [Quantum reinforcement learning](https://mlsanity.com/deeprl/quantum-reinforcement-learning/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Variational Quantum Algorithms](https://mlsanity.com/paper/2012.09265/) (2020)
- [Barren plateaus in quantum neural network training landscapes](https://mlsanity.com/paper/1803.11173/) (2018)
- [The power of quantum neural networks](https://mlsanity.com/paper/2011.00027/) (2020)
- [Variational Quantum Circuits for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.00397/) (2019)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Parametrized quantum policies for reinforcement learning](https://mlsanity.com/paper/2103.05577/) (2021)
- [Effect of barren plateaus on gradient-free optimization](https://mlsanity.com/paper/2011.12245/) (2020)
- [Does provable absence of barren plateaus imply classical simulability?](https://mlsanity.com/paper/2312.09121/) (2023)
- [On barren plateaus and cost function locality in variational quantum algorithms](https://mlsanity.com/paper/2011.10530/) (2020)
- [Trainability barriers and opportunities in quantum generative modeling](https://mlsanity.com/paper/2305.02881/) (2023)
- [Subtleties in the trainability of quantum machine learning models](https://mlsanity.com/paper/2110.14753/) (2021)
- [Reinforcement Learning Decoders for Fault-Tolerant Quantum Computation](https://mlsanity.com/paper/1810.07207/) (2018)
- [Quantum Deep Hedging](https://mlsanity.com/paper/2303.16585/) (2023)
- [Quantum Policy Gradient Algorithm with Optimized Action Decoding](https://mlsanity.com/paper/2212.06663/) (2022)
- [Policy Gradients using Variational Quantum Circuits](https://mlsanity.com/paper/2203.10591/) (2022)
- [Quantum policy gradient algorithms](https://mlsanity.com/paper/2212.09328/) (2022)
- [VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability](https://mlsanity.com/paper/2401.11555/) (2024)

## Built on by (most cited first, 4 of 4)

- [An Introduction to Quantum Reinforcement Learning (QRL)](https://mlsanity.com/paper/2409.05846/) (2024)
- [Dissecting Quantum Reinforcement Learning: A Systematic Evaluation of Key Components](https://mlsanity.com/paper/2511.17112/) (2025)
- [Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuits](https://mlsanity.com/paper/2605.03434/) (2026)
- [Formal Verification of Noisy Quantum Reinforcement Learning Policies](https://mlsanity.com/paper/2512.01502/) (2025)

## Abstract (arXiv)

This research explores the trainability of Parameterized Quantum circuit-based policies in Reinforcement Learning, an area that has recently seen a surge in empirical exploration. While some studies suggest improved sample complexity using quantum gradient estimation, the efficient trainability of these policies remains an open question. Our findings reveal significant challenges, including standard Barren Plateaus with exponentially small gradients and gradient explosion. These phenomena depend on the type of basis-state partitioning and mapping these partitions onto actions. For a polynomial number of actions, a trainable window can be ensured with a polynomial number of measurements if a contiguous-like partitioning of basis-states is employed. These results are empirically validated in a multi-armed bandit environment.