# Calibrated Model-Based Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1906.08312 · HTML: https://mlsanity.com/paper/1906.08312/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 30 of 31)

- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2412.14516/) (2024)
- [PAC Prediction Sets Under Covariate Shift](https://mlsanity.com/paper/2106.09848/) (2021)
- [The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2311.00168/) (2023)
- [Calibrated and Sharp Uncertainties in Deep Learning via Density Estimation](https://mlsanity.com/paper/2112.07184/) (2021)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching](https://mlsanity.com/paper/2306.14079/) (2023)
- [Modular Conformal Calibration](https://mlsanity.com/paper/2206.11468/) (2022)
- [Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.10369/) (2021)
- [Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?](https://mlsanity.com/paper/2107.11587/) (2021)
- [Online Calibrated and Conformal Prediction Improves Bayesian Optimization](https://mlsanity.com/paper/2112.04620/) (2021)
- [Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance](https://mlsanity.com/paper/2105.02331/) (2021)
- [Towards PAC Multi-Object Detection and Tracking](https://mlsanity.com/paper/2204.07482/) (2022)
- [Calibration of Continual Learning Models](https://mlsanity.com/paper/2404.07817/) (2024)
- [Zero-shot Model-based Reinforcement Learning using Large Language Models](https://mlsanity.com/paper/2410.11711/) (2024)
- [Model-based Offline Reinforcement Learning with Local Misspecification](https://mlsanity.com/paper/2301.11426/) (2023)
- [Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2306.17052/) (2023)
- [Calibrated Probabilistic Forecasts for Arbitrary Sequences](https://mlsanity.com/paper/2409.19157/) (2024)
- [DROMO: Distributionally Robust Offline Model-based Policy Optimization](https://mlsanity.com/paper/2109.07275/) (2021)
- [Approaching sales forecasting using recurrent neural networks and transformers](https://mlsanity.com/paper/2204.07786/) (2022)
- [A Verification Framework for Certifying Learning-Based Safety-Critical Aviation Systems](https://mlsanity.com/paper/2205.04590/) (2022)
- [Calibrated Regression Against An Adversary Without Regret](https://mlsanity.com/paper/2302.12196/) (2023)
- [Calibrated and Conformal Propensity Scores for Causal Effect Estimation](https://mlsanity.com/paper/2306.00382/) (2023)
- [Bellman Calibration for V-Learning in Offline Reinforcement Learning](https://mlsanity.com/paper/2512.23694/) (2025)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Calibrated Multivariate Distributional Regression with Pre-Rank Regularization](https://mlsanity.com/paper/2601.22895/) (2026)
- [MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planning](https://mlsanity.com/paper/2108.01295/) (2021)
- [Meta-learning to Calibrate Gaussian Processes with Deep Kernels for Regression Uncertainty Estimation](https://mlsanity.com/paper/2312.07952/) (2023)
- [Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees](https://mlsanity.com/paper/2604.14243/) (2026)

## Abstract (arXiv)

Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that good uncertainties must be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the textscHalfCheetah MuJoCo task, our system achieves state-of-the-art performance using 50% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.