# Replay-buffer engineering for noise-aware quantum circuit optimization

arXiv: https://arxiv.org/abs/2604.21863 · HTML: https://mlsanity.com/paper/2604.21863/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Quantum reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 71 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [Quantum reinforcement learning](https://mlsanity.com/deeprl/quantum-reinforcement-learning/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Variational Quantum Algorithms](https://mlsanity.com/paper/2012.09265/) (2020)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [Quantum circuit architecture search for variational quantum algorithms](https://mlsanity.com/paper/2010.10217/) (2020)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data](https://mlsanity.com/paper/2412.07762/) (2024)
- [Quantum Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2104.07715/) (2021)
- [Challenges and Opportunities in Quantum Optimization](https://mlsanity.com/paper/2312.02279/) (2023)
- [Curriculum reinforcement learning for quantum architecture search under hardware errors](https://mlsanity.com/paper/2402.03500/) (2024)
- [KANQAS: Kolmogorov-Arnold Network for Quantum Architecture Search](https://mlsanity.com/paper/2406.17630/) (2024)
- [Quantum Circuit Optimization with AlphaTensor](https://mlsanity.com/paper/2402.14396/) (2024)
- [Reinforcement Learning Decoders for Fault-Tolerant Quantum Computation](https://mlsanity.com/paper/1810.07207/) (2018)
- [Practical and efficient quantum circuit synthesis and transpiling with Reinforcement Learning](https://mlsanity.com/paper/2405.13196/) (2024)
- [Automated Quantum Circuit Design with Nested Monte Carlo Tree Search](https://mlsanity.com/paper/2207.00132/) (2022)
- [Prioritized Generative Replay](https://mlsanity.com/paper/2410.18082/) (2024)
- [Reinforcement learning-assisted quantum architecture search for variational quantum algorithms](https://mlsanity.com/paper/2402.13754/) (2024)
- [Physics-Informed Bayesian Optimization of Variational Quantum Circuits](https://mlsanity.com/paper/2406.06150/) (2024)
- [Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware](https://mlsanity.com/paper/2411.00230/) (2024)
- [TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search](https://mlsanity.com/paper/2505.09371/) (2025)
- [Reinforcement Learning for Quantum Technology](https://mlsanity.com/paper/2601.18953/) (2026)
- [Quantum Compiling with Reinforcement Learning on a Superconducting Processor](https://mlsanity.com/paper/2406.12195/) (2024)
- [Reliability-Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2506.18482/) (2025)
- [Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2512.10510/) (2025)

## Built on by (most cited first, 1 of 1)

- [Generative Replay Mitigates Sample Starvation in Quantum Architecture Search](https://mlsanity.com/paper/2609.11248/) (2026)

## Abstract (arXiv)

Deep reinforcement learning for quantum circuit optimization faces three bottlenecks: replay buffers that overlook temporal difference (TD) target reliability, curriculum-based architecture search requiring a full quantum-classical evaluation after every edit, and the discard of noiseless trajectories when retraining under hardware noise. We address these limitations by treating replay as a central algorithmic lever. We introduce ReaPER+, an annealed replay rule that transitions from TD-error prioritization to reliability-aware sampling as value estimates mature. ReaPER+ achieves up to 4x higher sample efficiency than fixed PER, ReaPER, and uniform replay, while matching prior on-policy solution quality with up to 32x fewer interactions At 12 qubits, fixed ReaPER reaches the lowest energy error in the fewest steps, while PER and uniform replay find more compact circuits at higher error. On tasks scaling to 20 qubits, ReaPER+ retains its advantage, demonstrating that reliability-aware annealing extends beyond small-system benchmarks. LunarLander-v3 confirms that the ReaPER+ is domain-agnostic, it improves success rates by up to 26.8% over PER and 21.8% over fixed ReaPER, with a 3% AUC gain over both. We further introduce OptCRLQAS, which amortizes quantum-classical evaluations across multiple architectural edits, reducing training wall-clock time by up to 67.5% on 12-qubit without degrading solution quality. Finally, lightweight replay-buffer transfer warm-starts noisy optimization from noiseless trajectories, without weight transfer or ε-greedy pretraining, reducing steps to chemical accuracy by 85-90% and final energy error by up to 90% relative to from-scratch learning. Transfer gains increase with system size. Together, these results establish experience storage, sampling, and transfer as decisive levers for sample efficient, noise-aware quantum circuit optimization.