mlsanityData to 14 Oct 2026

Diminishing Return of Value Expansion Methods

arXiv 2412.20537 ↗First posted 29 Dec 2024cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Model-based reinforcement learning aims to increase sample efficiency, but the accuracy of dynamics models and the resulting compounding errors are often seen as key limitations. This paper empirically investigates potential sample efficiency gains from improved dynamics models in model-based value expansion methods. Our study reveals two key findings when using oracle dynamics models to eliminate compounding errors. First, longer rollout horizons enhance sample efficiency, but the improvements quickly diminish with each additional expansion step. Second, increased model accuracy only marginally improves sample efficiency compared to learned models with identical horizons. These diminishing returns in sample efficiency are particularly noteworthy when compared to model-free value expansion methods. These model-free algorithms achieve comparable performance without the computational overhead. Our results suggest that the limitation of model-based value expansion methods cannot be attributed to model accuracy. Although higher accuracy is beneficial, even perfect models do not provide unrivaled sample efficiency. Therefore, the bottleneck exists elsewhere. These results challenge the common assumption that model accuracy is the primary constraint in model-based reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 67 references that are in the map. First posted on arXiv 29 Dec 2024.

Builds on

40 of 67 references.

References · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
28 more
  1. 2019Hamiltonian Neural Networkscited by 712
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  5. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 212
  6. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  8. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  9. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  10. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  11. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  12. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  13. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  14. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  15. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  16. 2020Bidirectional Model-based Policy Optimizationcited by 38
  17. 2019Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systemscited by 33
  18. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 30
  19. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  20. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 28
  21. 2020Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learningcited by 27
  22. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  23. 2021Learning Dynamics Models for Model Predictive Agentscited by 22
  24. 2022Active Exploration for Robotic Manipulationcited by 10
  25. 2020Dynamic Horizon Value Estimation for Model-based Reinforcement Learningcited by 8
  26. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 6
  27. 2021An Empirical Analysis of Measure-Valued Derivatives for Policy Gradientscited by 3
  28. 2022Revisiting Model-based Value Expansioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Limited Representational Power of Value Functions and its Links to Statistical (In)Efficiency
  2. 2026On Training in Imagination
  3. 2024Why long model-based rollouts are no reason for bad Q-value estimates
  4. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  5. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  6. 2025Scalable Offline Model-Based RL with Action Chunks
  7. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning
  8. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learning