# Adversarial Policies Beat Superhuman Go AIs

arXiv: https://arxiv.org/abs/2211.00241 · HTML: https://mlsanity.com/paper/2211.00241/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 86 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Adversarial Attacks on Neural Network Policies](https://mlsanity.com/paper/1702.02284/) (2017)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Real World Games Look Like Spinning Tops](https://mlsanity.com/paper/2004.09468/) (2020)
- [Accelerating Self-Play Learning in Go](https://mlsanity.com/paper/1902.10565/) (2019)
- [Challenges and Countermeasures for Adversarial Attacks on Deep Reinforcement Learning](https://mlsanity.com/paper/2001.09684/) (2020)
- [From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization](https://mlsanity.com/paper/2002.08456/) (2020)
- [ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero](https://mlsanity.com/paper/1902.04522/) (2019)
- [CIFAR10 to Compare Visual Recognition Performance between Deep Neural Networks and Humans](https://mlsanity.com/paper/1811.07270/) (2018)
- [Reducing Exploitability with Population Based Training](https://mlsanity.com/paper/2208.05083/) (2022)
- [Imitating Opponent to Win: Adversarial Policy Imitation Learning in Two-player Competitive Games](https://mlsanity.com/paper/2210.16915/) (2022)

## Built on by (most cited first, 29 of 29)

- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2402.03741/) (2024)
- [Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learning](https://mlsanity.com/paper/2302.00797/) (2023)
- [Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Models](https://mlsanity.com/paper/2507.06466/) (2025)
- [UNIDOOR: A Universal Framework for Action-Level Backdoor Attacks in Deep Reinforcement Learning](https://mlsanity.com/paper/2501.15529/) (2025)
- [Multi-Agent Diagnostics for Robustness via Illuminated Diversity](https://mlsanity.com/paper/2401.13460/) (2024)
- [TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models](https://mlsanity.com/paper/2506.12815/) (2025)
- [Minimax Exploiter: A Data Efficient Approach for Competitive Self-Play](https://mlsanity.com/paper/2311.17190/) (2023)
- [The Structural Safety Generalization Problem](https://mlsanity.com/paper/2504.09712/) (2025)
- [Look-ahead Reasoning with a Learned Model in Imperfect Information Games](https://mlsanity.com/paper/2510.05048/) (2025)
- [On existence, uniqueness and scalability of adversarial robustness measures for AI classifiers](https://mlsanity.com/paper/2310.14421/) (2023)
- [Responsible Emergent Multi-Agent Behavior](https://mlsanity.com/paper/2311.01609/) (2023)
- [Demystifying MuZero Planning: Interpreting the Learned Model](https://mlsanity.com/paper/2411.04580/) (2024)
- [Beyond Accuracy: A Geometric Stability Analysis of Large Language Models in Chess Evaluation](https://mlsanity.com/paper/2512.15033/) (2025)
- [Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learning](https://mlsanity.com/paper/2605.14587/) (2026)
- [Games of Knightian Uncertainty as AGI testbeds](https://mlsanity.com/paper/2406.18178/) (2024)
- [Bridging Local and Global Knowledge via Transformer in Board Games](https://mlsanity.com/paper/2410.05347/) (2024)
- [Search-contempt: a hybrid MCTS algorithm for training AlphaZero-like engines with better computational efficiency](https://mlsanity.com/paper/2504.07757/) (2025)
- [LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoning](https://mlsanity.com/paper/2507.15521/) (2025)
- [Relevance-Zone Reduction in Game Solving](https://mlsanity.com/paper/2510.00689/) (2025)
- [Outbidding and Outbluffing Elite Humans: Mastering Liar's Poker via Self-Play and Reinforcement Learning](https://mlsanity.com/paper/2511.03724/) (2025)
- [A Study of Solving Life-and-Death Problems in Go Using Relevance-Zone Based Solvers](https://mlsanity.com/paper/2512.21365/) (2025)
- [Nemobot Games: Crafting Strategic AI Gaming Agents for Interactive Learning with Large Language Models](https://mlsanity.com/paper/2604.21896/) (2026)
- [Synthetic Counteradaptation: A Principle of Human-AI Co-evolution](https://mlsanity.com/paper/2606.15503/) (2026)
- [Unsound Search with Policy and Value Networks in Legends of Code and Magic](https://mlsanity.com/paper/2609.06816/) (2026)

## Abstract (arXiv)

We attack the state-of-the-art Go-playing AI system KataGo by training adversarial policies against it, achieving a >97% win rate against KataGo running at superhuman settings. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders. Our attack transfers zero-shot to other superhuman Go-playing AIs, and is comprehensible to the extent that human experts can implement it without algorithmic assistance to consistently beat superhuman AIs. The core vulnerability uncovered by our attack persists even in KataGo agents adversarially trained to defend against our attack. Our results demonstrate that even superhuman AI systems may harbor surprising failure modes. Example games are available https://goattack.far.ai/.