mlsanityData to 14 Oct 2026

On Rollouts in Model-Based Reinforcement Learning

arXiv 2501.16918 ↗First posted 28 Jan 2025cs.LGcited by 10

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Model-based reinforcement learning (MBRL) seeks to enhance data efficiency by learning a model of the environment and generating synthetic rollouts from it. However, accumulated model errors during these rollouts can distort the data distribution, negatively impacting policy learning and hindering long-term planning. Thus, the accumulation of model errors is a key bottleneck in current MBRL methods. We propose Infoprop, a model-based rollout mechanism that separates aleatoric from epistemic model uncertainty and reduces the influence of the latter on the data distribution. Further, Infoprop keeps track of accumulated model errors along a model rollout and provides termination criteria to limit data corruption. We demonstrate the capabilities of Infoprop in the Infoprop-Dyna algorithm, reporting state-of-the-art performance in Dyna-style MBRL on common MuJoCo benchmark tasks while substantially increasing rollout length and data quality.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 54 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

38 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption29 May
  2. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning1 Mar

Other references · 36

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  4. 2023Mastering Diverse Domains through World Modelscited by 1,253
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2020Mastering Atari with Discrete World Modelscited by 1,007
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  12. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
24 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  2. 2018Understanding the impact of entropy on policy optimizationcited by 176
  3. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  4. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  5. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  6. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  7. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
  8. 2021Reinforcement Learning, Bit by Bitcited by 59
  9. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 54
  10. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  11. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  12. 2020Bidirectional Model-based Policy Optimizationcited by 38
  13. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  14. 2020Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learningcited by 26
  15. 2020Learning Accurate Long-term Dynamics for Model-based Reinforcement Learningcited by 21
  16. 2024A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismocited by 15
  17. 2017Prediction and Control with Temporal Segment Modelscited by 14
  18. 2022On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learningcited by 13
  19. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  20. 2018Towards a Simple Approach to Multi-step Model-based Reinforcement Learningcited by 9
  21. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  22. 2023Sample-Efficient Learning to Solve a Real-World Labyrinth Game Using Data-Augmented Model-Based Reinforcement Learningcited by 8
  23. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  24. 2014Kalman meets Shannoncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2026Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Modelscited by 5
  2. 2026WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Controlcited by 3
  3. 2025Sample-Efficient Reinforcement Learning of Koopman eNMPCcited by 2
  4. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  5. 2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
  6. 2026Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamicscited by 1
  7. 2026Learning to Race in Minutes: Infoprop Dyna on the Mini Wheelbotcited by 1
  8. 2026Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modelingcited by 0
  9. 2026All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learningcited by 0
  10. 2026Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Distribution Imagination for Model-Based Offline Reinforcement Learning
  2. 2026GRPODropout: Less is More for Online Reinforcement Learning Rollouts
  3. 2026On Training in Imagination
  4. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions
  5. 2026Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents
  6. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  7. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  8. 2024Why long model-based rollouts are no reason for bad Q-value estimates