# Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2009.09593 · HTML: https://mlsanity.com/paper/2009.09593/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 6 of its 45 references that are in the map.
- First posted on arXiv 21 Sep 2020.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 6 of 6)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance](https://mlsanity.com/paper/1812.02765/) (2018)
- [Bidirectional Model-based Policy Optimization](https://mlsanity.com/paper/2007.01995/) (2020)
- [Selective Dyna-style Planning Under Limited Model Capacity](https://mlsanity.com/paper/2007.02418/) (2020)

## Built on by (most cited first, 8 of 8)

- [UncAD: Towards Safe End-to-end Autonomous Driving via Online Map Uncertainty](https://mlsanity.com/paper/2504.12826/) (2025)
- [Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.09418/) (2022)
- [Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2303.03955/) (2023)
- [Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models](https://mlsanity.com/paper/2604.25416/) (2026)
- [Revisiting Model-based Value Expansion](https://mlsanity.com/paper/2203.14660/) (2022)
- [Segmenting Action-Value Functions Over Time-Scales in SARSA via TD(Delta)](https://mlsanity.com/paper/2411.14783/) (2024)
- [Diminishing Return of Value Expansion Methods](https://mlsanity.com/paper/2412.20537/) (2024)
- [A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic](https://mlsanity.com/paper/2603.28971/) (2026)

## Abstract (arXiv)

Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potential to harm the learning process. In this paper, we investigate the idea of using the model knowledge for value expansion adaptively. We propose a novel method called Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model usage with different rollout horizons. Inspired by reconstruction-based techniques that can be applied for visual data novelty detection, we utilize a world model with a reconstruction module for image feature extraction, in order to acquire more precise value estimation. The raw and the reconstructed images are both used to determine the appropriate horizon for adaptive value expansion. On several benchmark visual control tasks, experimental results show that DMVE outperforms all baselines in sample efficiency and final performance, indicating that DMVE can achieve more effective and accurate value estimation than state-of-the-art model-based methods.