mlsanityData to 14 Oct 2026

Mastering the Game of Go with Self-play Experience Replay

arXiv 2601.03306 ↗First posted 6 Jan 2026cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

The game of Go has long served as a benchmark for artificial intelligence, demanding sophisticated strategic reasoning and long-term planning. Previous approaches such as AlphaGo and its successors, have predominantly relied on model-based Monte-Carlo Tree Search (MCTS). In this work, we present QZero, a novel model-free reinforcement learning algorithm that forgoes search during training and learns a Nash equilibrium policy through self-play and off-policy experience replay. Built upon entropy-regularized Q-learning, QZero utilizes a single Q-value network to unify policy evaluation and improvement. Starting tabula rasa without human data and trained for 5 months with modest compute resources (7 GPUs), QZero achieved a performance level comparable to that of AlphaGo. This demonstrates, for the first time, the efficiency of using model-free reinforcement learning to master the game of Go, as well as the feasibility of off-policy reinforcement learning in solving large-scale and complex environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 27 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

14 of 27 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  11. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  12. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
2 more
  1. 2019ELF OpenGo: An Analysis and Open Reimplementation of AlphaZerocited by 39
  2. 2019Soft Q Networkcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations
  2. 2024A Survey on Self-play Methods in Reinforcement Learning
  3. 2025Zero-sum turn games using Q-learning: finite computation with security guarantees
  4. 2025OptionZero: Planning with Learned Options
  5. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games
  6. 2020A Game Theoretic Framework for Model Based Reinforcement Learning
  7. 2024Deep Reinforcement Learning for 5*5 Multiplayer Go
  8. 2025Relative Entropy Pathwise Policy Optimization