# Challenges for Reinforcement Learning in Quantum Circuit Design

arXiv: https://arxiv.org/abs/2312.11337 · HTML: https://mlsanity.com/paper/2312.11337/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 65 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [Quantum Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2104.07715/) (2021)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Mirror Learning: A Unifying Framework of Policy Optimisation](https://mlsanity.com/paper/2201.02373/) (2022)
- [Topological Quantum Compiling with Reinforcement Learning](https://mlsanity.com/paper/2004.04743/) (2020)
- [A Domain-agnostic, Noise-resistant, Hardware-efficient Evolutionary Variational Quantum Eigensolver](https://mlsanity.com/paper/1910.09694/) (2019)
- [Quantum Circuit Design Search](https://mlsanity.com/paper/2012.04046/) (2020)
- [Guide Your Agent with Adaptive Multimodal Rewards](https://mlsanity.com/paper/2309.10790/) (2023)
- [ProjectQ: An Open Source Software Framework for Quantum Computing](https://mlsanity.com/paper/1612.08091/) (2016)
- [Circuit Transformations for Quantum Architectures](https://mlsanity.com/paper/1902.09102/) (2019)
- [qgym: A Gym for Training and Benchmarking RL-Based Quantum Compilation](https://mlsanity.com/paper/2308.02536/) (2023)
- [Theta-Resonance: A Single-Step Reinforcement Learning Method for Design Space Exploration](https://mlsanity.com/paper/2211.02052/) (2022)
- [CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processing](https://mlsanity.com/paper/2304.13616/) (2023)

## Built on by (most cited first, 9 of 9)

- [Reinforcement learning-based architecture search for quantum machine learning](https://mlsanity.com/paper/2406.02717/) (2024)
- [Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware](https://mlsanity.com/paper/2411.00230/) (2024)
- [Evaluating Mutation Techniques in Genetic Algorithm-Based Quantum Circuit Synthesis](https://mlsanity.com/paper/2504.06413/) (2025)
- [Leveraging Diffusion Models for Parameterized Quantum Circuit Generation](https://mlsanity.com/paper/2505.20863/) (2025)
- [HQCC: A Hybrid Quantum-Classical Classifier with Adaptive Structure](https://mlsanity.com/paper/2504.02167/) (2025)
- [Automatic re-calibration of quantum devices by reinforcement learning](https://mlsanity.com/paper/2404.10726/) (2024)
- [Quantum Architecture Search for Solving Quantum Machine Learning Tasks](https://mlsanity.com/paper/2509.11198/) (2025)
- [Generative Quantum Data Embeddings for Supervised Learning](https://mlsanity.com/paper/2605.30866/) (2026)
- [Approximate Quantum State Preparation Through Proximal Policy Optimization](https://mlsanity.com/paper/2607.21121/) (2026)

## Abstract (arXiv)

Quantum computing (QC) in the current NISQ era is still limited in size and precision. Hybrid applications mitigating those shortcomings are prevalent to gain early insight and advantages. Hybrid quantum machine learning (QML) comprises both the application of QC to improve machine learning (ML) and ML to improve QC architectures. This work considers the latter, leveraging reinforcement learning (RL) to improve quantum circuit design (QCD), which we formalize by a set of generic objectives. Furthermore, we propose qcd-gym, a concrete framework formalized as a Markov decision process, to enable learning policies capable of controlling a universal set of continuously parameterized quantum gates. Finally, we provide benchmark comparisons to assess the shortcomings and strengths of current state-of-the-art RL algorithms.