The Surprising Difficulty of Search in Model-Based Reinforcement Learning

arXiv 2601.21306 ↗First posted 29 Jan 2026cs.LG cs.AIcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a drop-in replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating overestimation bias matters more than improving model or value function accuracy. Building on this insight, we identify that taking the minimum over an ensemble of value functions effectively addresses this bias and enables effective search, achieving state-of-the-art performance across multiple popular benchmark domains.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 57 of its 73 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

57 of 73 references.

Closest ancestors · 5

Same direction, newest first

Other references · 52

2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2015Continuous control with deep reinforcement learningcited by 3,407
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2018World Modelscited by 1,513
2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
2023Mastering Diverse Domains through World Modelscited by 1,253
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
2018DeepMind Control Suitecited by 956

38 more

2018Recurrent World Models Facilitate Policy Evolutioncited by 785
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
2022Temporal Difference Learning for Model Predictive Controlcited by 397
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
2021Mastering Atari Games with Limited Datacited by 237
2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
2019When to use parametric models in reinforcement learning?cited by 118
2017Value Prediction Networkcited by 117
2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 113
2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
2016The Predictron: End-To-End Learning and Planningcited by 93
2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
2017Robust Locally-Linear Controllable Embeddingcited by 38
2022Simplicial Embeddings in Self-Supervised Learning and Downstream Classificationcited by 24
2025Temporal Difference Flowscited by 24
2020Learning Accurate Long-term Dynamics for Model-based Reinforcement Learningcited by 21
2025TD-M(PC)2: Improving Temporal Difference MPC Through Policy Constraintcited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
2026EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Controlcited by 2
2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimizationcited by 0
2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
2026GEM-MPC: Balancing Exploration and Exploitation through Expert-Guided Planningcited by 0
2026VAMPS: Visual and Motor Policies from Sampling-Based Planningcited by 0

Similar, not cited

Close in wording, no citation either way