# Model-based Offline Quantum Reinforcement Learning

arXiv: https://arxiv.org/abs/2404.10017 · HTML: https://mlsanity.com/paper/2404.10017/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 14 Apr 2024.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [The effect of data encoding on the expressive power of variational quantum machine learning models](https://mlsanity.com/paper/2008.08605/) (2020)
- [Generalization in quantum machine learning from few training data](https://mlsanity.com/paper/2111.05292/) (2021)
- [Variational Quantum Circuits for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.00397/) (2019)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Reinforcement Learning with Quantum Variational Circuits](https://mlsanity.com/paper/2008.07524/) (2020)
- [A Survey on Quantum Reinforcement Learning](https://mlsanity.com/paper/2211.03464/) (2022)
- [Overcoming Model Bias for Robust Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2008.05533/) (2020)
- [Quantum Policy Iteration via Amplitude Estimation and Grover Search -- Towards Quantum Advantage for Reinforcement Learning](https://mlsanity.com/paper/2206.04741/) (2022)
- [Uncovering Instabilities in Variational-Quantum Deep Q-Networks](https://mlsanity.com/paper/2202.05195/) (2022)
- [Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.05433/) (2022)
- [Interpretable Control by Reinforcement Learning](https://mlsanity.com/paper/2007.09964/) (2020)
- [Behavior Constraining in Weight Space for Offline Reinforcement Learning](https://mlsanity.com/paper/2107.05479/) (2021)
- [BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading](https://mlsanity.com/paper/2305.00905/) (2023)

## Built on by (most cited first, 4 of 4)

- [Benchmarking Quantum Reinforcement Learning](https://mlsanity.com/paper/2501.15893/) (2025)
- [Quantum Artificial Intelligence: A Brief Survey](https://mlsanity.com/paper/2408.10726/) (2024)
- [From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware](https://mlsanity.com/paper/2509.07614/) (2025)
- [Staged Hybridisation for Visual Quantum Reinforcement Learning via Knowledge Distillation](https://mlsanity.com/paper/2606.30520/) (2026)

## Abstract (arXiv)

This paper presents the first algorithm for model-based offline quantum reinforcement learning and demonstrates its functionality on the cart-pole benchmark. The model and the policy to be optimized are each implemented as variational quantum circuits. The model is trained by gradient descent to fit a pre-recorded data set. The policy is optimized with a gradient-free optimization scheme using the return estimate given by the model as the fitness function. This model-based approach allows, in principle, full realization on a quantum computer during the optimization phase and gives hope that a quantum advantage can be achieved as soon as sufficiently powerful quantum computers are available.