mlsanityData to 7 Oct 2026

Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning

arXiv 2303.03955 ↗First posted 7 Mar 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning is one approach to increase sample efficiency. However, the accuracy of the dynamics model and the resulting compounding error over modelled trajectories are commonly regarded as key limitations. A natural question to ask is: How much more sample efficiency can be gained by improving the learned dynamics models? Our paper empirically answers this question for the class of model-based value expansion methods in continuous control problems. Value expansion methods should benefit from increased model accuracy by enabling longer rollout horizons and better value function approximations. Our empirical study, which leverages oracle dynamics models to avoid compounding model errors, shows that (1) longer horizons increase sample efficiency, but the gain in improvement decreases with each additional expansion step, and (2) the increased model accuracy only marginally increases the sample efficiency compared to learned models with identical horizons. Therefore, longer horizons and increased model accuracy yield diminishing returns in terms of sample efficiency. These improvements in sample efficiency are particularly disappointing when compared to model-free value expansion methods. Even though they introduce no computational overhead, we find their performance to be on-par with model-based value expansion methods. Therefore, we conclude that the limitation of model-based value expansion methods is not the model accuracy of the learned models. While higher model accuracy is beneficial, our experiments show that even a perfect model will not provide an un-rivalled sample efficiency but that the bottleneck lies elsewhere.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 61 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

36 of 61 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2019Hamiltonian Neural Networkscited by 702
24 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  3. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  4. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 205
  5. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  6. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  7. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  8. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  9. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  10. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  11. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  12. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 43
  13. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  14. 2020Bidirectional Model-based Policy Optimizationcited by 38
  15. 2019Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systemscited by 32
  16. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  17. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 28
  18. 2020Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learningcited by 27
  19. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  20. 2021Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learningcited by 25
  21. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  22. 2022Active Exploration for Robotic Manipulationcited by 9
  23. 2020Dynamic Horizon Value Estimation for Model-based Reinforcement Learningcited by 8
  24. 2021An Empirical Analysis of Measure-Valued Derivatives for Policy Gradientscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  2. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  3. 2024Diminishing Return of Value Expansion Methodscited by 0
  4. 2025Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processescited by 0
  5. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0