mlsanityData to 7 Oct 2026

Why long model-based rollouts are no reason for bad Q-value estimates

arXiv 2407.11751 ↗First posted 16 Jul 2024cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This paper explores the use of model-based offline reinforcement learning with long model rollouts. While some literature criticizes this approach due to compounding errors, many practitioners have found success in real-world applications. The paper aims to demonstrate that long rollouts do not necessarily result in exponentially growing errors and can actually produce better Q-value estimates than model-free methods. These findings can potentially enhance reinforcement learning techniques.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 8 of its 12 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

8 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Learning Control Policies for Variable Objectives from Offline Data11 Aug
  2. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning14 Jan
  3. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inference6 Feb

Other references · 5

  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 40
  3. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  4. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
  5. 2020Forethought and Hindsight in Credit Assignmentcited by 16

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Is Q-learning an Ill-posed Problem?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  2. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  3. 2023A Survey on Offline Model-Based Reinforcement Learning
  4. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
  5. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  6. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
  7. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
  8. 2023Model-based Offline Reinforcement Learning with Local Misspecification