mlsanityData to 7 Oct 2026

Muesli: Combining Improvements in Policy Optimization

arXiv 2104.06159 ↗First posted 13 Apr 2021cs.LG cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 54 of its 102 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

54 of 102 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2016OpenAI Gymcited by 1,684
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2020Mastering Atari with Discrete World Modelscited by 990
38 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  10. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  11. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  12. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  13. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  14. 2016Value Iteration Networkscited by 158
  15. 2018Meta-Gradient Reinforcement Learningcited by 156
  16. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  17. 2020Phasic Policy Gradientcited by 121
  18. 2019When to use parametric models in reinforcement learning?cited by 118
  19. 2017Value Prediction Networkcited by 117
  20. 2020Mirror Descent Policy Optimizationcited by 102
  21. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  22. 2016The Predictron: End-To-End Learning and Planningcited by 92
  23. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  24. 2016Learning values across many orders of magnitudecited by 80
  25. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  26. 2019An investigation of model-free planningcited by 64
  27. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  28. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  29. 2018Neural Predictive Belief Representationscited by 39
  30. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  31. 2020Model-based Reinforcement Learning: A Surveycited by 39
  32. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  33. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  34. 2020Causally Correct Partial Models for Reinforcement Learningcited by 24
  35. 2019On Inductive Biases in Deep Reinforcement Learningcited by 22
  36. 2017Learning model-based planning from scratchcited by 21
  37. 2019Combining Q-Learning and Search with Amortized Value Estimatescited by 19
  38. 2018ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Searchcited by 12

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 10; 2023: 16; 2024: 9; 2025: 4; 2026: 2.

53 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2022A Generalist Agentcited by 802
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  5. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  7. 2025Training Agents Inside of Scalable World Modelscited by 122
  8. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  9. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  10. 2023Vision-Language Models as a Source of Rewardscited by 65
  11. 2021Bootstrapped Meta-Learningcited by 45
  12. 2021Proper Value Equivalencecited by 33
38 more
  1. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  2. 2022Human-level Atari 200x fastercited by 28
  3. 2022You May Not Need Ratio Clipping in PPOcited by 19
  4. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  5. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  6. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  7. 2023MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Gamescited by 14
  8. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  9. 2021A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutionscited by 12
  10. 2022Generalized Data Distribution Iterationcited by 12
  11. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  12. 2025A Clean Slate for Offline Reinforcement Learningcited by 10
  13. 2024Efficient Multi-agent Reinforcement Learning by Planningcited by 9
  14. 2024FactorSim: Generative Simulation via Factorized Representationcited by 9
  15. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8
  16. 2021GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learningcited by 7
  17. 2021Self-Consistent Models and Valuescited by 7
  18. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  19. 2023What model does MuZero learn?cited by 7
  20. 2023RL-X: A Deep Reinforcement Learning Library (not only) for RoboCupcited by 7
  21. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  22. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  23. 2024Value Improved Actor Critic Algorithmscited by 6
  24. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  25. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  26. 2021GrowSpace: Learning How to Shape Plantscited by 2
  27. 2022DNA: Proximal Policy Optimization with a Dual Network Architecturecited by 2
  28. 2024Reinforcement Learning for Sustainable Energy: A Surveycited by 2
  29. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  30. 2025OptionZero: Planning with Learned Optionscited by 2
  31. 2025Safe In-Context Reinforcement Learningcited by 2
  32. 2021Model-Advantage and Value-Aware Models for Model-Based Reinforcement Learning: Bridging the Gap in Theory and Practicecited by 1
  33. 2022Graph Backup: Data Efficient Backup Exploiting Markovian Transitionscited by 1
  34. 2023Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learningcited by 1
  35. 2023Trust-Region-Free Policy Optimization for Stochastic Policiescited by 1
  36. 2023Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methodscited by 1
  37. 2023Hierarchical Reinforcement Learning in Complex 3D Environmentscited by 0
  38. 2023Acceleration in Policy Optimizationcited by 0

50 most cited of 53 All 53 (JSON)