# Adaptive Rollout Length for Model-Based RL Using Model-Free Deep RL

arXiv: https://arxiv.org/abs/2206.02380 · HTML: https://mlsanity.com/paper/2206.02380/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Emotion in Reinforcement Learning Agents and Robots: A Survey](https://mlsanity.com/paper/1705.05172/) (2017)
- [The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces](https://mlsanity.com/paper/1806.01825/) (2018)

## Built on by (most cited first, 5 of 5)

- [Truncating Trajectories in Monte Carlo Reinforcement Learning](https://mlsanity.com/paper/2305.04361/) (2023)
- [Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach](https://mlsanity.com/paper/2410.13463/) (2024)
- [Confidence-Controlled Exploration: Efficient Sparse-Reward Policy Learning for Robot Navigation](https://mlsanity.com/paper/2306.06192/) (2023)
- [Scalable Reinforcement Learning via Adaptive Batch Scaling](https://mlsanity.com/paper/2605.21557/) (2026)
- [EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management](https://mlsanity.com/paper/2609.27547/) (2026)

## Abstract (arXiv)

Model-based reinforcement learning promises to learn an optimal policy from fewer interactions with the environment compared to model-free reinforcement learning by learning an intermediate model of the environment in order to predict future interactions. When predicting a sequence of interactions, the rollout length, which limits the prediction horizon, is a critical hyperparameter as accuracy of the predictions diminishes in the regions that are further away from real experience. As a result, with a longer rollout length, an overall worse policy is learned in the long run. Thus, the hyperparameter provides a trade-off between quality and efficiency. In this work, we frame the problem of tuning the rollout length as a meta-level sequential decision-making problem that optimizes the final policy learned by model-based reinforcement learning given a fixed budget of environment interactions by adapting the hyperparameter dynamically based on feedback from the learning process, such as accuracy of the model and the remaining budget of interactions. We use model-free deep reinforcement learning to solve the meta-level decision problem and demonstrate that our approach outperforms common heuristic baselines on two well-known reinforcement learning environments.