mlsanityData to 7 Oct 2026

Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

arXiv 2107.11587 ↗First posted 24 Jul 2021cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models using a fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them on the control problem. Using this framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot by two to four folds, using an aggressive training schedule which is outside of the hyperparameter interval usually considered

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 24 Jul 2021.

Builds on

28 of 56 references.

References · 28

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Variational Inference with Normalizing Flowscited by 2,161
  3. 2016Density estimation using Real NVPcited by 2,029
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2016OpenAI Gymcited by 1,684
  6. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  10. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  11. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  12. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
16 more
  1. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  2. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  3. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  4. 2020Model-Based Offline Planningcited by 126
  5. 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 97
  6. 2019Exploring Model-based Planning with Policy Networkscited by 90
  7. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  8. 2013RNADE: The real-valued neural autoregressive density-estimatorcited by 65
  9. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  10. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  11. 2018A survey on policy search algorithms for learning robot controllers in a handful of trialscited by 33
  12. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  13. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  14. 2019Supervised Anomaly Detection based on Deep Autoregressive Density Estimatorscited by 3
  15. 2017Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learningcited by 2
  16. 2016Micro-Data Learning: The Other End of the Spectrumcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 1; 2024: 4; 2025: 1.

11 papers.

  1. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  2. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  3. 2024A call for embodied AIcited by 7
  4. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  5. 2021Which Model to Trust: Assessing the Influence of Models on the Performance of Reinforcement Learning Algorithms for Continuous Control Taskscited by 3
  6. 2022Guided Safe Shooting: model based reinforcement learning with safety constraintscited by 1
  7. 2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
  8. 2024Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learningcited by 1
  9. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learningcited by 1
  10. 2021Knothe-Rosenblatt transport for Unsupervised Domain Adaptationcited by 0
  11. 2025From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimationcited by 0