# VQC-Based Reinforcement Learning with Data Re-uploading: Performance and Trainability

arXiv: https://arxiv.org/abs/2401.11555 · HTML: https://mlsanity.com/paper/2401.11555/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Quantum reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 21 Jan 2024.

## Placed in

- Deep reinforcement learning methods / [Quantum reinforcement learning](https://mlsanity.com/deeprl/quantum-reinforcement-learning/)

## Builds on (most cited first, 17 of 17)

- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Variational Quantum Algorithms](https://mlsanity.com/paper/2012.09265/) (2020)
- [Barren plateaus in quantum neural network training landscapes](https://mlsanity.com/paper/1803.11173/) (2018)
- [The effect of data encoding on the expressive power of variational quantum machine learning models](https://mlsanity.com/paper/2008.08605/) (2020)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Connecting ansatz expressibility to gradient magnitudes and barren plateaus](https://mlsanity.com/paper/2101.02138/) (2021)
- [Variational Quantum Circuits for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.00397/) (2019)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Does provable absence of barren plateaus imply classical simulability?](https://mlsanity.com/paper/2312.09121/) (2023)
- [Reinforcement Learning with Quantum Variational Circuits](https://mlsanity.com/paper/2008.07524/) (2020)
- [The Born Supremacy: Quantum Advantage and Training of an Ising Born Machine](https://mlsanity.com/paper/1904.02214/) (2019)
- [A Survey on Quantum Reinforcement Learning](https://mlsanity.com/paper/2211.03464/) (2022)
- [Theoretical Guarantees for Permutation-Equivariant Quantum Neural Networks](https://mlsanity.com/paper/2210.09974/) (2022)
- [A survey of dimensionality reduction techniques](https://mlsanity.com/paper/1403.2877/) (2014)
- [Escaping from the Barren Plateau via Gaussian Initializations in Deep Variational Quantum Circuits](https://mlsanity.com/paper/2203.09376/) (2022)
- [Equivariant quantum circuits for learning on weighted graphs](https://mlsanity.com/paper/2205.06109/) (2022)
- [Policy Gradients using Variational Quantum Circuits](https://mlsanity.com/paper/2203.10591/) (2022)

## Built on by (most cited first, 12 of 12)

- [Investigating and Mitigating Barren Plateaus in Variational Quantum Circuits: A Survey](https://mlsanity.com/paper/2407.17706/) (2024)
- [Benchmarking Quantum Reinforcement Learning](https://mlsanity.com/paper/2501.15893/) (2025)
- [Evolutionary Optimization for Designing Variational Quantum Circuits with High Model Capacity](https://mlsanity.com/paper/2412.12484/) (2024)
- [Learning to Program Quantum Measurements for Machine Learning](https://mlsanity.com/paper/2505.13525/) (2025)
- [Data re-uploading in Quantum Machine Learning for time series: application to traffic forecasting](https://mlsanity.com/paper/2501.12776/) (2025)
- [Trainability issues in quantum policy gradients](https://mlsanity.com/paper/2406.09614/) (2024)
- [Quantum Hamiltonian Embedding of Images for Data Reuploading Classifiers](https://mlsanity.com/paper/2407.14055/) (2024)
- [Introduction to Quantum Machine Learning and Quantum Architecture Search](https://mlsanity.com/paper/2504.16131/) (2025)
- [Hamiltonian-based Quantum Reinforcement Learning for Neural Combinatorial Optimization](https://mlsanity.com/paper/2405.07790/) (2024)
- [Dissecting Quantum Reinforcement Learning: A Systematic Evaluation of Key Components](https://mlsanity.com/paper/2511.17112/) (2025)
- [Dynamic Spectrum Access for Ambient Backscatter Communication-assisted D2D Systems with Quantum Reinforcement Learning](https://mlsanity.com/paper/2410.17971/) (2024)
- [Entanglement as a Structural Complexity Axis: A PAC-Bayesian View of Generalization in Quantum Policies and Value Functions](https://mlsanity.com/paper/2607.06230/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) consists of designing agents that make intelligent decisions without human supervision. When used alongside function approximators such as Neural Networks (NNs), RL is capable of solving extremely complex problems. Deep Q-Learning, a RL algorithm that uses Deep NNs, achieved super-human performance in some specific tasks. Nonetheless, it is also possible to use Variational Quantum Circuits (VQCs) as function approximators in RL algorithms. This work empirically studies the performance and trainability of such VQC-based Deep Q-Learning models in classic control benchmark environments. More specifically, we research how data re-uploading affects both these metrics. We show that the magnitude and the variance of the gradients of these models remain substantial throughout training due to the moving targets of Deep Q-Learning. Moreover, we empirically show that increasing the number of qubits does not lead to an exponential vanishing behavior of the magnitude and variance of the gradients for a PQC approximating a 2-design, unlike what was expected due to the Barren Plateau Phenomenon. This hints at the possibility of VQCs being specially adequate for being used as function approximators in such a context.