mlsanityData to 7 Oct 2026

Diminishing Return of Value Expansion Methods

arXiv 2412.20537 ↗First posted 29 Dec 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning aims to increase sample efficiency, but the accuracy of dynamics models and the resulting compounding errors are often seen as key limitations. This paper empirically investigates potential sample efficiency gains from improved dynamics models in model-based value expansion methods. Our study reveals two key findings when using oracle dynamics models to eliminate compounding errors. First, longer rollout horizons enhance sample efficiency, but the improvements quickly diminish with each additional expansion step. Second, increased model accuracy only marginally improves sample efficiency compared to learned models with identical horizons. These diminishing returns in sample efficiency are particularly noteworthy when compared to model-free value expansion methods. These model-free algorithms achieve comparable performance without the computational overhead. Our results suggest that the limitation of model-based value expansion methods cannot be attributed to model accuracy. Although higher accuracy is beneficial, even perfect models do not provide unrivaled sample efficiency. Therefore, the bottleneck exists elsewhere. These results challenge the common assumption that model accuracy is the primary constraint in model-based reinforcement learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 67 references that are in the map. First posted on arXiv 29 Dec 2024.

Builds on

40 of 67 references.

References · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
28 more
  1. 2019Hamiltonian Neural Networkscited by 702
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  4. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  5. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 205
  6. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  8. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  9. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  10. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  11. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  12. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  13. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  14. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 43
  15. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  16. 2020Bidirectional Model-based Policy Optimizationcited by 38
  17. 2019Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systemscited by 32
  18. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  19. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 28
  20. 2020Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learningcited by 27
  21. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  22. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  23. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  24. 2022Active Exploration for Robotic Manipulationcited by 9
  25. 2020Dynamic Horizon Value Estimation for Model-based Reinforcement Learningcited by 8
  26. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  27. 2021An Empirical Analysis of Measure-Valued Derivatives for Policy Gradientscited by 3
  28. 2022Revisiting Model-based Value Expansioncited by 3

Built on by

Not cited yet.