# Deep Reinforcement Learning with Quantum-inspired Experience Replay

arXiv: https://arxiv.org/abs/2101.02034 · HTML: https://mlsanity.com/paper/2101.02034/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 56 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 17 of 17)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Efficient Learning for Deep Quantum Neural Networks](https://mlsanity.com/paper/1902.10445/) (2019)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)
- [Quantum-enhanced machine learning](https://mlsanity.com/paper/1610.08251/) (2016)
- [Remember and Forget for Experience Replay](https://mlsanity.com/paper/1807.05827/) (2018)
- [Competitive Experience Replay](https://mlsanity.com/paper/1902.00528/) (2019)
- [Reinforcement Learning Using Quantum Boltzmann Machines](https://mlsanity.com/paper/1612.05695/) (2016)
- [Fidelity-based Probabilistic Q-learning for Control of Quantum Systems](https://mlsanity.com/paper/1806.03145/) (2018)
- [Multiqubit and multilevel quantum reinforcement learning with quantum technologies](https://mlsanity.com/paper/1709.07848/) (2017)
- [A Quantum-inspired Similarity Measure for the Analysis of Complete Weighted Graphs](https://mlsanity.com/paper/1904.13239/) (2019)
- [Basic protocols in quantum reinforcement learning with superconducting circuits](https://mlsanity.com/paper/1701.05131/) (2017)

## Built on by (most cited first, 19 of 19)

- [A Survey on Quantum Reinforcement Learning](https://mlsanity.com/paper/2211.03464/) (2022)
- [Quantum Deep Reinforcement Learning for Robot Navigation Tasks](https://mlsanity.com/paper/2202.12180/) (2022)
- [Variational Quantum Soft Actor-Critic](https://mlsanity.com/paper/2112.11921/) (2021)
- [Efficient Diversity-based Experience Replay for Deep Reinforcement Learning](https://mlsanity.com/paper/2410.20487/) (2024)
- [Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments](https://mlsanity.com/paper/2207.01251/) (2022)
- [Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversity](https://mlsanity.com/paper/2305.04180/) (2023)
- [Quantum framework for Reinforcement Learning: Integrating Markov decision process, quantum arithmetic, and trajectory search](https://mlsanity.com/paper/2412.18208/) (2024)
- [Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07729/) (2022)
- [Auxiliary Task-based Deep Reinforcement Learning for Quantum Control](https://mlsanity.com/paper/2302.14312/) (2023)
- [Self-evolving Autoencoder Embedded Q-Network](https://mlsanity.com/paper/2402.11604/) (2024)
- [Quantum Adjoint Convolutional Layers for Effective Data Representation](https://mlsanity.com/paper/2404.17378/) (2024)
- [A Comprehensively Adaptive Architectural Optimization-Ingrained Quantum Neural Network Model for Cloud Workloads Prediction](https://mlsanity.com/paper/2507.08317/) (2025)
- [Scalable Quantum Reinforcement Learning on NISQ Devices with Dynamic-Circuit Qubit Reuse and Grover Optimization](https://mlsanity.com/paper/2509.16002/) (2025)
- [Improving Experience Replay through Modeling of Similar Transitions' Sets](https://mlsanity.com/paper/2111.06907/) (2021)
- [AI-Assisted Authentication: State of the Art, Taxonomy and Future Roadmap](https://mlsanity.com/paper/2204.12492/) (2022)
- [Towards Quantum-Enabled 6G Slicing](https://mlsanity.com/paper/2212.11755/) (2022)
- [Adaptive BESS and Grid Setpoints Optimization: A Model-Free Framework for Efficient Battery Management under Dynamic Tariff Pricing](https://mlsanity.com/paper/2408.09989/) (2024)
- [Extending a Quantum Reinforcement Learning Exploration Policy with Flags to Connect Four](https://mlsanity.com/paper/2505.04371/) (2025)
- [QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2510.27091/) (2025)

## Abstract (arXiv)

In this paper, a novel training paradigm inspired by quantum computation is proposed for deep reinforcement learning (DRL) with experience replay. In contrast to traditional experience replay mechanism in DRL, the proposed deep reinforcement learning with quantum-inspired experience replay (DRL-QER) adaptively chooses experiences from the replay buffer according to the complexity and the replayed times of each experience (also called transition), to achieve a balance between exploration and exploitation. In DRL-QER, transitions are first formulated in quantum representations, and then the preparation operation and the depreciation operation are performed on the transitions. In this progress, the preparation operation reflects the relationship between the temporal difference errors (TD-errors) and the importance of the experiences, while the depreciation operation is taken into account to ensure the diversity of the transitions. The experimental results on Atari 2600 games show that DRL-QER outperforms state-of-the-art algorithms such as DRL-PER and DCRL on most of these games with improved training efficiency, and is also applicable to such memory-based DRL approaches as double network and dueling network.