# BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading

arXiv: https://arxiv.org/abs/2305.00905 · HTML: https://mlsanity.com/paper/2305.00905/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [The power of quantum neural networks](https://mlsanity.com/paper/2011.00027/) (2020)
- [The effect of data encoding on the expressive power of variational quantum machine learning models](https://mlsanity.com/paper/2008.08605/) (2020)
- [Generalization in quantum machine learning from few training data](https://mlsanity.com/paper/2111.05292/) (2021)
- [Variational Quantum Circuits for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.00397/) (2019)
- [Benchmarking Batch Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1910.01708/) (2019)
- [A Survey on Quantum Reinforcement Learning](https://mlsanity.com/paper/2211.03464/) (2022)
- [An Optimistic Perspective on Offline Reinforcement Learning](https://mlsanity.com/paper/1907.04543/) (2019)
- [Input Redundancy for Parameterized Quantum Circuits](https://mlsanity.com/paper/1901.11434/) (2019)
- [Hybrid quantum-classical classifier based on tensor network and variational quantum circuit](https://mlsanity.com/paper/2011.14651/) (2020)
- [Quantum Policy Gradient Algorithm with Optimized Action Decoding](https://mlsanity.com/paper/2212.06663/) (2022)
- [A scale-dependent notion of effective dimension](https://mlsanity.com/paper/2001.10872/) (2020)
- [Uncovering Instabilities in Variational-Quantum Deep Q-Networks](https://mlsanity.com/paper/2202.05195/) (2022)
- [Incremental Data-Uploading for Full-Quantum Classification](https://mlsanity.com/paper/2205.03057/) (2022)
- [Quantum Multi-Agent Actor-Critic Neural Networks for Internet-Connected Multi-Robot Coordination in Smart Factory Management](https://mlsanity.com/paper/2301.04012/) (2023)

## Built on by (most cited first, 3 of 3)

- [Model-based Offline Quantum Reinforcement Learning](https://mlsanity.com/paper/2404.10017/) (2024)
- [Guided-SPSA: Simultaneous Perturbation Stochastic Approximation assisted by the Parameter Shift Rule](https://mlsanity.com/paper/2404.15751/) (2024)
- [Robustness and Generalization in Quantum Reinforcement Learning via Lipschitz Regularization](https://mlsanity.com/paper/2410.21117/) (2024)

## Abstract (arXiv)

Deep reinforcement learning (DRL) often requires a large number of data and environment interactions, making the training process time-consuming. This challenge is further exacerbated in the case of batch RL, where the agent is trained solely on a pre-collected dataset without environment interactions. Recent advancements in quantum computing suggest that quantum models might require less data for training compared to classical methods. In this paper, we investigate this potential advantage by proposing a batch RL algorithm that utilizes VQC as function approximators within the discrete batch-constraint deep Q-learning (BCQ) algorithm. Additionally, we introduce a novel data re-uploading scheme by cyclically shifting the order of input variables in the data encoding layers. We evaluate the efficiency of our algorithm on the OpenAI CartPole environment and compare its performance to the classical neural network-based discrete BCQ.