mlsanityData to 7 Oct 2026

Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

arXiv 1803.00101 ↗First posted 28 Feb 2018cs.LG cs.AI stat.MLcited by 152

Not on the map. Why?

Abstract (arXiv)

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 37; 2022: 25; 2023: 27; 2024: 25; 2025: 20; 2026: 15.

152 papers.

Sort:

By area:

  1. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  2. 2021Learning to drive from a world on railscited by 114
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2021Gradients are Not All You Needcited by 82
  5. 2022A Survey on Model-based Reinforcement Learningcited by 65
  6. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  7. 2021Learning Task Informed Abstractionscited by 58
  8. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  9. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  10. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  11. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  12. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
38 more
  1. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  2. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  3. 2021Model-Based Opponent Modelingcited by 21
  4. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  5. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  6. 2022A Survey on Reinforcement Learning in Aviation Applicationscited by 19
  7. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  8. 2021Model-Invariant State Abstractions for Model-Based Reinforcement Learningcited by 18
  9. 2024Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulationcited by 18
  10. 2021Value Iteration in Continuous Actions, States and Timecited by 17
  11. 2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17
  12. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  13. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 17
  14. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
  15. 2024PWM: Policy Learning with Multi-Task World Modelscited by 17
  16. 2022Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learningcited by 16
  17. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimizationcited by 16
  18. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  19. 2020Bridging Imagination and Reality for Model-Based Deep Reinforcement Learningcited by 15
  20. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  21. 2021Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Modelcited by 14
  22. 2021MHER: Model-based Hindsight Experience Replaycited by 14
  23. 2022When to Update Your Model: Constrained Model-based Reinforcement Learningcited by 14
  24. 2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
  25. 2025Generative Models in Decision Making: A Surveycited by 14
  26. 2021MapGo: Model-Assisted Policy Optimization for Goal-Oriented Taskscited by 13
  27. 2021High-Accuracy Model-Based Reinforcement Learning, a Surveycited by 13
  28. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  29. 2023Dynamically Conservative Self-Driving Planner for Long-Tail Casescited by 13
  30. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  31. 2024Prioritized Generative Replaycited by 13
  32. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  33. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  34. 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learningcited by 11
  35. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  36. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  37. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10
  38. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10

50 most cited of 152 All 152 (JSON)