# Reinforcement learning-assisted quantum architecture search for variational quantum algorithms

arXiv: https://arxiv.org/abs/2402.13754 · HTML: https://mlsanity.com/paper/2402.13754/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Quantum reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 173 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [Quantum reinforcement learning](https://mlsanity.com/deeprl/quantum-reinforcement-learning/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Recurrent Quantum Neural Networks](https://mlsanity.com/paper/2006.14619/) (2020)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [Quantum Graph Neural Networks](https://mlsanity.com/paper/1909.12264/) (2019)
- [Quantum Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2104.07715/) (2021)
- [Curriculum reinforcement learning for quantum architecture search under hardware errors](https://mlsanity.com/paper/2402.03500/) (2024)
- [Quantum Architecture Search via Continual Reinforcement Learning](https://mlsanity.com/paper/2112.05779/) (2021)
- [Reinforcement Learning Assisted Recursive QAOA](https://mlsanity.com/paper/2207.06294/) (2022)

## Built on by (most cited first, 10 of 10)

- [KANQAS: Kolmogorov-Arnold Network for Quantum Architecture Search](https://mlsanity.com/paper/2406.17630/) (2024)
- [Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware](https://mlsanity.com/paper/2411.00230/) (2024)
- [TensorRL-QAS: Reinforcement learning with tensor networks for improved quantum architecture search](https://mlsanity.com/paper/2505.09371/) (2025)
- [Improving thermal state preparation of Sachdev-Ye-Kitaev model with reinforcement learning on quantum hardware](https://mlsanity.com/paper/2501.11454/) (2025)
- [BenchRL-QAS: Benchmarking reinforcement learning algorithms for quantum architecture search](https://mlsanity.com/paper/2507.12189/) (2025)
- [Self-Modulating Quantum Fast-Weight Programmers for Efficient Adaptive Sequential Learning](https://mlsanity.com/paper/2606.24933/) (2026)
- [Enhancing Quantum Memory Lifetime with Measurement-Free Local Error Correction and Reinforcement Learning](https://mlsanity.com/paper/2408.09524/) (2024)
- [Replay-buffer engineering for noise-aware quantum circuit optimization](https://mlsanity.com/paper/2604.21863/) (2026)
- [AQMLator -- An Auto Quantum Machine Learning E-Platform](https://mlsanity.com/paper/2409.18338/) (2024)
- [Quantum Architecture Search for Solving Quantum Machine Learning Tasks](https://mlsanity.com/paper/2509.11198/) (2025)

## Abstract (arXiv)

A significant hurdle in the noisy intermediate-scale quantum (NISQ) era is identifying functional quantum circuits. These circuits must also adhere to the constraints imposed by current quantum hardware limitations. Variational quantum algorithms (VQAs), a class of quantum-classical optimization algorithms, were developed to address these challenges in the currently available quantum devices. However, the overall performance of VQAs depends on the initialization strategy of the variational circuit, the structure of the circuit (also known as ansatz), and the configuration of the cost function. Focusing on the structure of the circuit, in this thesis, we improve the performance of VQAs by automating the search for an optimal structure for the variational circuits using reinforcement learning (RL). Within the thesis, the optimality of a circuit is determined by evaluating its depth, the overall count of gates and parameters, and its accuracy in solving the given problem. The task of automating the search for optimal quantum circuits is known as quantum architecture search (QAS). The majority of research in QAS is primarily focused on a noiseless scenario. Yet, the impact of noise on the QAS remains inadequately explored. In this thesis, we tackle the issue by introducing a tensor-based quantum circuit encoding, restrictions on environment dynamics to explore the search space of possible circuits efficiently, an episode halting scheme to steer the agent to find shorter circuits, a double deep Q-network (DDQN) with an ε-greedy policy for better stability. The numerical experiments on noiseless and noisy quantum hardware show that in dealing with various VQAs, our RL-based QAS outperforms existing QAS. Meanwhile, the methods we propose in the thesis can be readily adapted to address a wide range of other VQAs.