Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We consider the problem of offline reinforcement learning where only a set of system transitions is made available for policy optimization. Following recent advances in the field, we consider a model-based reinforcement learning algorithm that infers the system dynamics from the available data and performs policy optimization on imaginary model rollouts. This approach is vulnerable to exploiting model errors which can lead to catastrophic failures on the real system. The standard solution is to rely on ensembles for uncertainty heuristics and to avoid exploiting the model where it is too uncertain. We challenge the popular belief that we must resort to ensembles by showing that better performance can be obtained with a single well-calibrated autoregressive model on the D4RL benchmark. We also analyze static metrics of model-learning and conclude on the important model properties for the final performance of the agent.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 68 references that are in the map. First posted on arXiv 5 Feb 2024.
Builds on
46 of 68 references.
Closest ancestors · 10
Same direction, newest first
- 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization15 Sep
- 2021Model-Based Offline Planning with Trajectory Pruning16 May
- 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
- 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
- 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 36
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2020Mastering Atari with Discrete World Modelscited by 1,007
- 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
- 2016Pixel Recurrent Neural Networkscited by 816
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
24 more
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
- 2017Masked Autoregressive Flow for Density Estimationcited by 684
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
- 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
- 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
- 2019Benchmarking Model-Based Reinforcement Learningcited by 162
- 2020Model-Based Offline Planningcited by 129
- 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
- 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
- 2021Epistemic Neural Networkscited by 120
- 2016Neural Autoregressive Distribution Estimationcited by 95
- 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 95
- 2019Exploring Model-based Planning with Policy Networkscited by 90
- 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
- 2013RNADE: The real-valued neural autoregressive density-estimatorcited by 66
- 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
- 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
- 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
- 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
- 2021Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?cited by 11
Built on by
Citing papers per year: 2024: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2018Model-Ensemble Trust-Region Policy Optimization
- 2024Offline Model-Based Reinforcement Learning with Anti-Exploration
- 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
- 2023A Survey on Offline Model-Based Reinforcement Learning
- 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
- 2020MOPO: Model-based Offline Policy Optimization
- 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
- 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning