# EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data

arXiv: https://arxiv.org/abs/2403.00564 · HTML: https://mlsanity.com/paper/2403.00564/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 28 of 28)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes](https://mlsanity.com/paper/2211.11744/) (2022)
- [Online and Offline Reinforcement Learning by Planning with a Learned Model](https://mlsanity.com/paper/2104.06294/) (2021)
- [Learning and Planning in Complex Action Spaces](https://mlsanity.com/paper/2104.06303/) (2021)
- [Multi-step Reinforcement Learning: A Unifying Algorithm](https://mlsanity.com/paper/1703.01327/) (2017)
- [DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training](https://mlsanity.com/paper/2305.12127/) (2023)

## Built on by (most cited first, 30 of 40)

- [Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control](https://mlsanity.com/paper/2405.16158/) (2024)
- [Towards General-Purpose Model-Free Reinforcement Learning](https://mlsanity.com/paper/2501.16142/) (2025)
- [Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos](https://mlsanity.com/paper/2502.09886/) (2025)
- [Learning Transformer-based World Models with Contrastive Predictive Coding](https://mlsanity.com/paper/2503.04416/) (2025)
- [Bootstrapped Model Predictive Control](https://mlsanity.com/paper/2503.18871/) (2025)
- [A Step Toward World Models: A Survey on Robotic Manipulation](https://mlsanity.com/paper/2511.02097/) (2025)
- [Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining](https://mlsanity.com/paper/2410.00564/) (2024)
- [R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation](https://mlsanity.com/paper/2603.18202/) (2026)
- [Multi-Task Reinforcement Learning Enables Parameter Scaling](https://mlsanity.com/paper/2503.05126/) (2025)
- [Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC](https://mlsanity.com/paper/2411.03820/) (2024)
- [Hadamax Encoding: Elevating Performance in Model-Free Atari](https://mlsanity.com/paper/2505.15345/) (2025)
- [Accurate and Efficient World Modeling with Masked Latent Transformers](https://mlsanity.com/paper/2507.04075/) (2025)
- [Simulus: Combining Improvements in Sample-Efficient World Model Agents](https://mlsanity.com/paper/2502.11537/) (2025)
- [Trust-Region Twisted Policy Improvement](https://mlsanity.com/paper/2504.06048/) (2025)
- [TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance](https://mlsanity.com/paper/2509.26627/) (2025)
- [The Surprising Difficulty of Search in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2601.21306/) (2026)
- [AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models](https://mlsanity.com/paper/2505.24784/) (2025)
- [Reliability-Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2506.18482/) (2025)
- [World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications](https://mlsanity.com/paper/2606.00133/) (2026)
- [Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease](https://mlsanity.com/paper/2507.06326/) (2025)
- [A KL-regularization Framework for Learning to Plan with Adaptive Priors](https://mlsanity.com/paper/2510.04280/) (2025)
- [Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach](https://mlsanity.com/paper/2510.23216/) (2025)
- [Planning in Branch-and-Bound: Model-Based Reinforcement Learning for Exact Combinatorial Optimization](https://mlsanity.com/paper/2511.09219/) (2025)
- [Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2602.10044/) (2026)
- [What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty](https://mlsanity.com/paper/2603.02491/) (2026)
- [Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Prediction](https://mlsanity.com/paper/2603.07083/) (2026)
- [EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Control](https://mlsanity.com/paper/2605.16692/) (2026)
- [Object-Centric World Models Meet Monte Carlo Tree Search](https://mlsanity.com/paper/2601.06604/) (2026)
- [Reason to Play: Behavioral and Brain Alignment Between Frontier LRMs and Human Game Learners](https://mlsanity.com/paper/2605.08019/) (2026)
- [PriorZero: Bridging Language Priors and World Models for Decision Making](https://mlsanity.com/paper/2605.12289/) (2026)

## Abstract (arXiv)

Sample efficiency remains a crucial challenge in applying Reinforcement Learning (RL) to real-world tasks. While recent algorithms have made significant strides in improving sample efficiency, none have achieved consistently superior performance across diverse domains. In this paper, we introduce EfficientZero V2, a general framework designed for sample-efficient RL algorithms. We have expanded the performance of EfficientZero to multiple domains, encompassing both continuous and discrete actions, as well as visual and low-dimensional inputs. With a series of improvements we propose, EfficientZero V2 outperforms the current state-of-the-art (SOTA) by a significant margin in diverse tasks under the limited data setting. EfficientZero V2 exhibits a notable advancement over the prevailing general algorithm, DreamerV3, achieving superior outcomes in 50 of 66 evaluated tasks across diverse benchmarks, such as Atari 100k, Proprio Control, and Vision Control.