# Decoding surface codes with deep reinforcement learning and probabilistic policy reuse

arXiv: https://arxiv.org/abs/2212.11890 · HTML: https://mlsanity.com/paper/2212.11890/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 114 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation](https://mlsanity.com/paper/1807.01473/) (2018)
- [Reinforcement learning for optimization of variational quantum circuit architectures](https://mlsanity.com/paper/2103.16089/) (2021)
- [Quantum Architecture Search via Deep Reinforcement Learning](https://mlsanity.com/paper/2104.07715/) (2021)
- [Universal Quantum Control through Deep Reinforcement Learning](https://mlsanity.com/paper/1803.01857/) (2018)
- [DisCoRL: Continual Reinforcement Learning via Policy Distillation](https://mlsanity.com/paper/1907.05855/) (2019)
- [Deep Transformer Q-Networks for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2206.01078/) (2022)
- [Optimizing Quantum Error Correction Codes with Reinforcement Learning](https://mlsanity.com/paper/1812.08451/) (2018)
- [Reinforcement Learning Decoders for Fault-Tolerant Quantum Computation](https://mlsanity.com/paper/1810.07207/) (2018)
- [Topological Quantum Compiling with Reinforcement Learning](https://mlsanity.com/paper/2004.04743/) (2020)
- [Quantum Architecture Search via Continual Reinforcement Learning](https://mlsanity.com/paper/2112.05779/) (2021)
- [Quantum Neural Architecture Search with Quantum Circuits Metric and Bayesian Optimization](https://mlsanity.com/paper/2206.14115/) (2022)
- [Automated Quantum Circuit Design with Nested Monte Carlo Tree Search](https://mlsanity.com/paper/2207.00132/) (2022)
- [Deep neural decoders for near term fault-tolerant experiments](https://mlsanity.com/paper/1802.06441/) (2018)
- [Variational quantum compiling with double Q-learning](https://mlsanity.com/paper/2103.11611/) (2021)
- [Monte Carlo Tree Search based Hybrid Optimization of Variational Quantum Circuits](https://mlsanity.com/paper/2203.16707/) (2022)
- [Efficient and practical quantum compiler towards multi-qubit systems with deep reinforcement learning](https://mlsanity.com/paper/2204.06904/) (2022)

## Built on by (most cited first, 1 of 1)

- [Learning to Decode the Surface Code with a Recurrent, Transformer-Based Neural Network](https://mlsanity.com/paper/2310.05900/) (2023)

## Abstract (arXiv)

Quantum computing (QC) promises significant advantages on certain hard computational tasks over classical computers. However, current quantum hardware, also known as noisy intermediate-scale quantum computers (NISQ), are still unable to carry out computations faithfully mainly because of the lack of quantum error correction (QEC) capability. A significant amount of theoretical studies have provided various types of QEC codes; one of the notable topological codes is the surface code, and its features, such as the requirement of only nearest-neighboring two-qubit control gates and a large error threshold, make it a leading candidate for scalable quantum computation. Recent developments of machine learning (ML)-based techniques especially the reinforcement learning (RL) methods have been applied to the decoding problem and have already made certain progress. Nevertheless, the device noise pattern may change over time, making trained decoder models ineffective. In this paper, we propose a continual reinforcement learning method to address these decoding challenges. Specifically, we implement double deep Q-learning with probabilistic policy reuse (DDQN-PPR) model to learn surface code decoding strategies for quantum environments with varying noise patterns. Through numerical simulations, we show that the proposed DDQN-PPR model can significantly reduce the computational complexity. Moreover, increasing the number of trained policies can further improve the agent's performance. Our results open a way to build more capable RL agents which can leverage previously gained knowledge to tackle QEC challenges.