mlsanityData to 7 Oct 2026

Online and Offline Reinforcement Learning by Planning with a Learned Model

arXiv 2104.06294 ↗First posted 13 Apr 2021cs.LGcited by 90

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning efficiently from small amounts of data has long been the focus of model-based reinforcement learning, both for the online case when interacting with the environment and the offline case when learning from a fixed dataset. However, to date no single unified algorithm could demonstrate state-of-the-art results in both settings. In this work, we describe the Reanalyse algorithm which uses model-based policy and value improvement operators to compute new improved training targets on existing data points, allowing efficient learning for data budgets varying by several orders of magnitude. We further show that Reanalyse can also be used to learn entirely from demonstrations without any environment interactions, as in the case of offline Reinforcement Learning (offline RL). Combining Reanalyse with the MuZero algorithm, we introduce MuZero Unplugged, a single unified algorithm for any data budget, including offline RL. In contrast to previous work, our algorithm does not require any special adaptations for the off-policy or offline RL settings. MuZero Unplugged sets new state-of-the-art results in the RL Unplugged offline RL benchmark as well as in the online RL benchmark of Atari in the standard 200 million frame setting.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 44 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

29 of 44 references.

Closest ancestors · 11

Same direction, newest first

  1. 2020POPO: Pessimistic Offline Policy Optimization26 Dec
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs18 Oct
  4. 2020Critic Regularized Regression26 Jun
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2016Layer Normalizationcited by 5,579
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2017Decoupled Weight Decay Regularizationcited by 1,722
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2019Solving Rubik's Cube with a Robot Handcited by 699
  12. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
6 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  2. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  3. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  4. 2020Model-Based Offline Planningcited by 126
  5. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  6. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 16; 2023: 26; 2024: 13; 2025: 15; 2026: 7.

90 papers.

Sort:

By area:

  1. 2021Mastering Atari Games with Limited Datacited by 235
  2. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  3. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  4. 2021The Benchmark Lotterycited by 133
  5. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  6. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  7. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  8. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  9. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  10. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  11. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  12. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
38 more
  1. 2021Proper Value Equivalencecited by 33
  2. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  3. 2022Human-level Atari 200x fastercited by 28
  4. 2022Large-Scale Retrieval for Reinforcement Learningcited by 24
  5. 2022Planning for Sample Efficient Imitation Learningcited by 20
  6. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  7. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  8. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  9. 2024Sample-Efficient Alignment for LLMscited by 16
  10. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  11. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  12. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 13
  13. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  14. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 12
  15. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  16. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
  17. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  18. 2024Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learningcited by 11
  19. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  20. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10
  21. 2024LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Neededcited by 10
  22. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  23. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8
  24. 2023Planning with Sequence Models through Iterative Energy Minimizationcited by 8
  25. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learningcited by 8
  26. 2021Self-Consistent Models and Valuescited by 7
  27. 2022Epistemic Monte Carlo Tree Searchcited by 7
  28. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  29. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithmscited by 6
  30. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  31. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  32. 2024Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstractioncited by 6
  33. 2024Value Improved Actor Critic Algorithmscited by 6
  34. 2025Bootstrap Off-policy with World Modelcited by 6
  35. 2022Design of experiments for the calibration of history-dependent models via deep reinforcement learning and an enhanced Kalman filtercited by 5
  36. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  37. 2023Investigating the role of model-based learning in exploration and transfercited by 5
  38. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  5. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  6. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  7. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning