mlsanityData to 7 Oct 2026

The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms

arXiv 2303.00694 ↗First posted 1 Mar 2023cs.LG cs.RO cs.SYcited by 12

Not on the map. Why?

Abstract (arXiv)

We propose a novel approach to addressing two fundamental challenges in Model-based Reinforcement Learning (MBRL): the computational expense of repeatedly finding a good policy in the learned model, and the objective mismatch between model fitting and policy computation. Our "lazy" method leverages a novel unified objective, Performance Difference via Advantage in Model, to capture the performance difference between the learned policy and expert policy under the true dynamics. This objective demonstrates that optimizing the expected policy advantage in the learned model under an exploration distribution is sufficient for policy computation, resulting in a significant boost in computational efficiency compared to traditional planning methods. Additionally, the unified objective uses a value moment matching term for model fitting, which is aligned with the model's usage during policy computation. We present two no-regret algorithms to optimize the proposed objective, and demonstrate their statistical and computational gains compared to existing MBRL methods through simulated benchmarks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 63 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

32 of 63 references.

References · 32

  1. 2016OpenAI Gymcited by 1,684
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2019Introduction to Online Convex Optimizationcited by 793
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
20 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  2. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  3. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  5. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  7. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  8. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  9. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  10. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  11. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  12. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  13. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  14. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  15. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  16. 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 36
  17. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  18. 2021Minimax Model Learningcited by 16
  19. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  20. 2020Planning and Execution using Inaccurate Models with Provable Guaranteescited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 6.

12 papers.

  1. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  2. 2024Hybrid Inverse Reinforcement Learningcited by 31
  3. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  4. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  5. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  6. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  7. 2024Value of Information and Reward Specification in Active Inference and POMDPscited by 4
  8. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  9. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  10. 2025Policy Testing in Markov Decision Processescited by 1
  11. 2024Multi-Agent Transfer Learning via Temporal Contrastive Learningcited by 0
  12. 2025Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processescited by 0