# Selective Dyna-style Planning Under Limited Model Capacity

arXiv: https://arxiv.org/abs/2007.02418 · HTML: https://mlsanity.com/paper/2007.02418/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2020.

## Built on by (most cited first, 30 of 30)

- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [Revisiting Design Choices in Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2110.04135/) (2021)
- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)
- [Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption](https://mlsanity.com/paper/2405.19014/) (2024)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [A Distributional Analogue to the Successor Representation](https://mlsanity.com/paper/2402.08530/) (2024)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?](https://mlsanity.com/paper/2107.11587/) (2021)
- [Dynamic Horizon Value Estimation for Model-based Reinforcement Learning](https://mlsanity.com/paper/2009.09593/) (2020)
- [Policy Gradients Incorporating the Future](https://mlsanity.com/paper/2108.02096/) (2021)
- [Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.09418/) (2022)
- [No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL](https://mlsanity.com/paper/2205.08716/) (2022)
- [Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2303.03955/) (2023)
- [Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR](https://mlsanity.com/paper/2605.05965/) (2026)
- [Should Models Be Accurate?](https://mlsanity.com/paper/2205.10736/) (2022)
- [Maximum Entropy Model Correction in Reinforcement Learning](https://mlsanity.com/paper/2311.17855/) (2023)
- [Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2406.04088/) (2024)
- [Calibrated Value-Aware Model Learning with Probabilistic Environment Models](https://mlsanity.com/paper/2505.22772/) (2025)
- [Memory Allocation in Resource-Constrained Reinforcement Learning](https://mlsanity.com/paper/2506.17263/) (2025)
- [Distillation Policy Optimization](https://mlsanity.com/paper/2302.00533/) (2023)
- [λ-models: Effective Decision-Aware Reinforcement Learning with Latent Models](https://mlsanity.com/paper/2306.17366/) (2023)
- [Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2405.03878/) (2024)
- [vMFER: Von Mises-Fisher Experience Resampling Based on Uncertainty of Gradient Directions for Policy Improvement](https://mlsanity.com/paper/2405.08638/) (2024)
- [Bounding-Box Inference for Error-Aware Model-Based Reinforcement Learning](https://mlsanity.com/paper/2406.16006/) (2024)
- [Why long model-based rollouts are no reason for bad Q-value estimates](https://mlsanity.com/paper/2407.11751/) (2024)
- [Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning](https://mlsanity.com/paper/2602.12375/) (2026)
- [Discerning Temporal Difference Learning](https://mlsanity.com/paper/2310.08091/) (2023)
- [Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planning](https://mlsanity.com/paper/2406.19561/) (2024)
- [Diminishing Return of Value Expansion Methods](https://mlsanity.com/paper/2412.20537/) (2024)
- [A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimization](https://mlsanity.com/paper/2510.01457/) (2025)

## Abstract (arXiv)

In model-based reinforcement learning, planning with an imperfect model of the environment has the potential to harm learning progress. But even when a model is imperfect, it may still contain information that is useful for planning. In this paper, we investigate the idea of using an imperfect model selectively. The agent should plan in parts of the state space where the model would be helpful but refrain from using the model where it would be harmful. An effective selective planning mechanism requires estimating predictive uncertainty, which arises out of aleatoric uncertainty, parameter uncertainty, and model inadequacy, among other sources. Prior work has focused on parameter uncertainty for selective planning. In this work, we emphasize the importance of model inadequacy. We show that heteroscedastic regression can signal predictive uncertainty arising from model inadequacy that is complementary to that which is detected by methods designed for parameter uncertainty, indicating that considering both parameter uncertainty and model inadequacy may be a more promising direction for effective selective planning than either in isolation.