# Visualizing MuZero Models

arXiv: https://arxiv.org/abs/2102.12924 · HTML: https://mlsanity.com/paper/2102.12924/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [Behaviour Suite for Reinforcement Learning](https://mlsanity.com/paper/1908.03568/) (2019)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [The Value Equivalence Principle for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2011.03506/) (2020)
- [The Predictron: End-To-End Learning and Planning](https://mlsanity.com/paper/1612.08810/) (2016)
- [Observe and Look Further: Achieving Consistent Performance on Atari](https://mlsanity.com/paper/1805.11593/) (2018)
- [Learning to Predict Without Looking Ahead: World Models Without Forward Prediction](https://mlsanity.com/paper/1910.13038/) (2019)
- [The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning](https://mlsanity.com/paper/2007.03158/) (2020)
- [Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey](https://mlsanity.com/paper/2008.05598/) (2020)

## Built on by (most cited first, 9 of 9)

- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [UniZero: Generalized and Efficient Planning with Scalable Latent World Models](https://mlsanity.com/paper/2406.10667/) (2024)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Trust-Region Twisted Policy Improvement](https://mlsanity.com/paper/2504.06048/) (2025)
- [OptionZero: Planning with Learned Options](https://mlsanity.com/paper/2502.16634/) (2025)
- [Demystifying MuZero Planning: Interpreting the Learned Model](https://mlsanity.com/paper/2411.04580/) (2024)
- [Assessing Policy, Loss and Planning Combinations in Reinforcement Learning using a New Modular Architecture](https://mlsanity.com/paper/2201.02874/) (2022)
- [Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RL](https://mlsanity.com/paper/2206.02039/) (2022)
- [Adaptable Hindsight Experience Replay for Search-Based Learning](https://mlsanity.com/paper/2511.03405/) (2025)

## Abstract (arXiv)

MuZero, a model-based reinforcement learning algorithm that uses a value equivalent dynamics model, achieved state-of-the-art performance in Chess, Shogi and the game of Go. In contrast to standard forward dynamics models that predict a full next state, value equivalent models are trained to predict a future value, thereby emphasizing value relevant information in the representations. While value equivalent models have shown strong empirical success, there is no research yet that visualizes and investigates what types of representations these models actually learn. Therefore, in this paper we visualize the latent representation of MuZero agents. We find that action trajectories may diverge between observation embeddings and internal state transition dynamics, which could lead to instability during planning. Based on this insight, we propose two regularization techniques to stabilize MuZero's performance. Additionally, we provide an open-source implementation of MuZero along with an interactive visualizer of learned representations, which may aid further investigation of value equivalent algorithms.