# Modeling quantum neural network gradient with reinforcement learning

arXiv: https://arxiv.org/abs/2609.31066 · HTML: https://mlsanity.com/paper/2609.31066/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Quantum reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Quantum reinforcement learning](https://mlsanity.com/deeprl/quantum-reinforcement-learning/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Barren plateaus in quantum neural network training landscapes](https://mlsanity.com/paper/1803.11173/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Noise-Induced Barren Plateaus in Variational Quantum Algorithms](https://mlsanity.com/paper/2007.14384/) (2020)
- [Barren Plateaus in Variational Quantum Computing](https://mlsanity.com/paper/2405.00781/) (2024)
- [QuantumNAS: Noise-Adaptive Search for Robust Quantum Circuits](https://mlsanity.com/paper/2107.10845/) (2021)
- [Layerwise learning for quantum neural networks](https://mlsanity.com/paper/2006.14904/) (2020)
- [Effect of barren plateaus on gradient-free optimization](https://mlsanity.com/paper/2011.12245/) (2020)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Quantum Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2104.07715/) (2021)
- [Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves](https://mlsanity.com/paper/2009.11243/) (2020)
- [Learning to learn with quantum neural networks via classical neural networks](https://mlsanity.com/paper/1907.05415/) (2019)
- [Escaping from the Barren Plateau via Gaussian Initializations in Deep Variational Quantum Circuits](https://mlsanity.com/paper/2203.09376/) (2022)
- [Learning to Optimize Variational Quantum Circuits to Solve Combinatorial Problems](https://mlsanity.com/paper/1911.11071/) (2019)
- [Quantum Machine Learning Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2407.20147/) (2024)
- [A Unified Theory of Quantum Neural Network Loss Landscapes](https://mlsanity.com/paper/2408.11901/) (2024)
- [Optimizing Quantum Variational Circuits with Deep Reinforcement Learning](https://mlsanity.com/paper/2109.03188/) (2021)
- [Reinforcement Learning for Variational Quantum Circuits Design](https://mlsanity.com/paper/2409.05475/) (2024)
- [A Study on Optimization Techniques for Variational Quantum Circuits in Reinforcement Learning](https://mlsanity.com/paper/2405.12354/) (2024)
- [Reinforcement-Learning-Based Variational Quantum Circuits Optimization for Combinatorial Problems](https://mlsanity.com/paper/1911.04574/) (2019)
- [TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search](https://mlsanity.com/paper/2505.09371/) (2025)

## Abstract (arXiv)

Training quantum neural networks (QNNs) on near-term hardware remains hampered by two compounding difficulties: the exponential vanishing of gradient variance known as the barren plateau, and the O(L · 2n) time and memory cost of differentiating through an n-qubit, L-layer circuit. We propose RLQ-Grad, a reinforcement-learning-based optimizer in which a classical policy πφ (a spectrally-normalized PPO agent) learns to propose parameter updates directly, conditioned on the QNN's current parameters, loss, accuracy, and previous update. Because the surrogate gradient is emitted by a classical network rather than obtained by differentiating through the unitary U(θ), its variance is not constrained by the barren plateau concentration bound, and its cost scales with the number of trainable parameters rather than the Hilbert-space dimension. We prove these properties formally and verify them on a hardware-efficient ansatz across four supervised benchmarks with up to n=20 qubits. RLQ-Grad preserves a near-flat gradient-variance curve where backpropagation, parameter-shift, and adjoint differentiation decay by 1 to 2 orders of magnitude. Accounting for the full training pipeline (PPO rollouts, actor-critic updates, and optimizer states), RLQ-Grad needs under 2 MB of memory and runs 2490×, 7876×, and 673× faster per iteration than these three methods at n=20. It improves top-1 accuracy by up to +10% over gradient-based baselines on circuits of up to 12 qubits, and matches dedicated barren plateau mitigation methods on CIFAR-10 at 14 to 20 qubits, where evolutionary and gradient-free optimizers collapse to chance.