# MuZero with Self-competition for Rate Control in VP9 Video Compression

arXiv: https://arxiv.org/abs/2202.06626 · HTML: https://mlsanity.com/paper/2202.06626/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Builds on (most cited first, 25 of 25)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Exploration-Exploitation in Constrained MDPs](https://mlsanity.com/paper/2003.02189/) (2020)
- [YouTube UGC Dataset for Video Compression Research](https://mlsanity.com/paper/1904.06457/) (2019)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Chip Placement with Deep Reinforcement Learning](https://mlsanity.com/paper/2004.10746/) (2020)
- [Value constrained model-free continuous control](https://mlsanity.com/paper/1902.04623/) (2019)
- [Ranked Reward: Enabling Self-Play Reinforcement Learning for Combinatorial Optimization](https://mlsanity.com/paper/1807.01672/) (2018)
- [Balancing Constraints and Rewards with Meta-Gradient D4PG](https://mlsanity.com/paper/2010.06324/) (2020)
- [Reward Constrained Interactive Recommendation with Natural Language Feedback](https://mlsanity.com/paper/2005.01618/) (2020)
- [A Dual-Critic Reinforcement Learning Framework for Frame-level Bit Allocation in HEVC/H.265](https://mlsanity.com/paper/2104.01735/) (2021)
- [Self-Play Learning Without a Reward Metric](https://mlsanity.com/paper/1912.07557/) (2019)
- [Neural Rate Control for Video Encoding using Imitation Learning](https://mlsanity.com/paper/2012.05339/) (2020)

## Built on by (most cited first, 28 of 28)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [Adversarial Training for High-Stakes Reliability](https://mlsanity.com/paper/2205.01663/) (2022)
- [Evaluating Long-Term Memory in 3D Mazes](https://mlsanity.com/paper/2210.13383/) (2022)
- [Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs](https://mlsanity.com/paper/2306.11700/) (2023)
- [Beyond Games: A Systematic Review of Neural Monte Carlo Tree Search Applications](https://mlsanity.com/paper/2303.08060/) (2023)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [GVFs in the Real World: Making Predictions Online for Water Treatment](https://mlsanity.com/paper/2312.01624/) (2023)
- [Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search](https://mlsanity.com/paper/2206.00702/) (2022)
- [Epistemic Monte Carlo Tree Search](https://mlsanity.com/paper/2210.13455/) (2022)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Hybrid Search for Efficient Planning with Completeness Guarantees](https://mlsanity.com/paper/2310.12819/) (2023)
- [Reinforcement Learning with a Terminator](https://mlsanity.com/paper/2205.15376/) (2022)
- [Equivariant MuZero](https://mlsanity.com/paper/2302.04798/) (2023)
- [Policy-Based Self-Competition for Planning Problems](https://mlsanity.com/paper/2306.04403/) (2023)
- [Policy Gradient With Serial Markov Chain Reasoning](https://mlsanity.com/paper/2210.06766/) (2022)
- [Real-time scheduling of renewable power systems through planning-based reinforcement learning](https://mlsanity.com/paper/2303.05205/) (2023)
- [Deep Video Codec Control for Vision Models](https://mlsanity.com/paper/2308.16215/) (2023)
- [Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security](https://mlsanity.com/paper/2401.10149/) (2024)
- [RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression](https://mlsanity.com/paper/2501.12216/) (2025)
- [OptionZero: Planning with Learned Options](https://mlsanity.com/paper/2502.16634/) (2025)
- [ReACT: Reinforcement Learning for Controller Parametrization using B-Spline Geometries](https://mlsanity.com/paper/2401.05251/) (2024)
- [A Perspective on Deep Vision Performance with Standard Image and Video Codecs](https://mlsanity.com/paper/2404.12330/) (2024)
- [Demystifying MuZero Planning: Interpreting the Learned Model](https://mlsanity.com/paper/2411.04580/) (2024)
- [Agents Explore the Environment Beyond Good Actions to Improve Their Model for Better Decisions](https://mlsanity.com/paper/2306.03408/) (2023)
- [Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs](https://mlsanity.com/paper/2408.10015/) (2024)
- [Resource-Aware Neural Network Pruning Using Graph-based Reinforcement Learning](https://mlsanity.com/paper/2509.10526/) (2025)
- [SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning](https://mlsanity.com/paper/2610.08076/) (2026)

## Abstract (arXiv)

Video streaming usage has seen a significant rise as entertainment, education, and business increasingly rely on online video. Optimizing video compression has the potential to increase access and quality of content to users, and reduce energy use and costs overall. In this paper, we present an application of the MuZero algorithm to the challenge of video compression. Specifically, we target the problem of learning a rate control policy to select the quantization parameters (QP) in the encoding process of libvpx, an open source VP9 video compression library widely used by popular video-on-demand (VOD) services. We treat this as a sequential decision making problem to maximize the video quality with an episodic constraint imposed by the target bitrate. Notably, we introduce a novel self-competition based reward mechanism to solve constrained RL with variable constraint satisfaction difficulty, which is challenging for existing constrained RL methods. We demonstrate that the MuZero-based rate control achieves an average 6.28% reduction in size of the compressed videos for the same delivered video quality level (measured as PSNR BD-rate) compared to libvpx's two-pass VBR rate control policy, while having better constraint satisfaction behavior.