mlsanityData to 7 Oct 2026

Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption

arXiv 2405.19014 ↗First posted 29 May 2024cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Dyna-style model-based reinforcement learning (MBRL) combines model-free agents with predictive transition models through model-based rollouts. This combination raises a critical question: 'When to trust your model?'; i.e., which rollout length results in the model providing useful data? Janner et al. (2019) address this question by gradually increasing rollout lengths throughout the training. While theoretically tempting, uniform model accuracy is a fallacy that collapses at the latest when extrapolating. Instead, we propose asking the question 'Where to trust your model?'. Using inherent model uncertainty to consider local accuracy, we obtain the Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption (MACURA) algorithm. We propose an easy-to-tune rollout mechanism and demonstrate substantial improvements in data efficiency and performance compared to state-of-the-art deep MBRL methods on the MuJoCo benchmark.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

43 of 57 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2018World Modelscited by 1,444
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2023Mastering Diverse Domains through World Modelscited by 1,193
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
31 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  5. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  6. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  7. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  8. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  9. 2019On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract meanscited by 61
  10. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  11. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
  12. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  13. 2020Bidirectional Model-based Policy Optimizationcited by 38
  14. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  15. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  16. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  17. 2021Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rolloutscited by 24
  18. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  19. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  20. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  21. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimizationcited by 16
  22. 2023Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learningcited by 16
  23. 2023Model-Based Uncertainty in Value Functionscited by 15
  24. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  25. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  26. 2021On-Policy Model Errors in Reinforcement Learningcited by 6
  27. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimizationcited by 6
  28. 2019Uncertainty-aware Model-based Policy Optimizationcited by 5
  29. 2023Data-efficient Deep Reinforcement Learning for Vehicle Trajectory Controlcited by 4
  30. 2020Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errorscited by 3
  31. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 11.

17 papers.

Sort:

By area:

  1. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  2. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  3. 2026Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Modelscited by 5
  4. 2026Robust Shielding for Safe Reinforcement Learningcited by 2
  5. 2025Sample-Efficient Reinforcement Learning of Koopman eNMPCcited by 1
  6. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  7. 2025Double Horizon Model-Based Policy Optimizationcited by 1
  8. 2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
  9. 2026Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamicscited by 1
  10. 2026Learning to Race in Minutes: Infoprop Dyna on the Mini Wheelbotcited by 1
  11. 2026ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPCcited by 1
  12. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0
5 more
  1. 2026IPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learningcited by 0
  2. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  3. 2026Soft MPCritic: Amortized Model Predictive Value Iterationcited by 0
  4. 2026All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learningcited by 0
  5. 2026Lucid Dreaming for World Models: Learning to Doubt Imagination and Decide by Trustcited by 0