mlsanityData to 14 Oct 2026

Double Horizon Model-Based Policy Optimization

arXiv 2512.15439 ↗First posted 17 Dec 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Model-based reinforcement learning (MBRL) reduces the cost of real-environment sampling by generating synthetic trajectories (called rollouts) from a learned dynamics model. However, choosing the length of the rollouts poses two dilemmas: (1) Longer rollouts better preserve on-policy training but amplify model bias, indicating the need for an intermediate horizon to mitigate distribution shift (i.e., the gap between on-policy and past off-policy samples). (2) Moreover, a longer model rollout may reduce value estimation bias but raise the variance of policy gradients due to backpropagation through multiple steps, implying another intermediate horizon for stable gradient estimates. However, these two optimal horizons may differ. To resolve this conflict, we propose Double Horizon Model-Based Policy Optimization (DHMBPO), which divides the rollout procedure into a long "distribution rollout" (DR) and a short "training rollout" (TR). The DR generates on-policy state samples for mitigating distribution shift. In contrast, the short TR leverages differentiable transitions to offer accurate value gradient estimation with stable gradient updates, thereby requiring fewer updates and reducing overall runtime. We demonstrate that the double-horizon approach effectively balances distribution shift, model bias, and gradient instability, and surpasses existing MBRL methods on continuous-control benchmarks in terms of both sample efficiency and runtime.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 40 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

24 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption29 May

Other references · 23

  1. 2016Layer Normalizationcited by 5,669
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
11 more
  1. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  2. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  3. 2019A Theory of Regularized Markov Decision Processescited by 286
  4. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  6. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  7. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  8. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  9. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  10. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  11. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Setscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024DyDiff: Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learning
  2. 2025First Order Model-Based RL through Decoupled Backpropagation
  3. 2025Improving DAPO from a Mixed-Policy Perspective
  4. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  5. 2025On Rollouts in Model-Based Reinforcement Learning
  6. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
  7. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO
  8. 2020Dynamic Horizon Value Estimation for Model-based Reinforcement Learning