mlsanityData to 14 Oct 2026

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning

arXiv 2412.14312 ↗First posted 18 Dec 2024cs.LGcited by 4

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Dyna-style off-policy model-based reinforcement learning (DMBRL) algorithms are a family of techniques for generating synthetic state transition data and thereby enhancing the sample efficiency of off-policy RL algorithms. This paper identifies and investigates a surprising performance gap observed when applying DMBRL algorithms across different benchmark environments with proprioceptive observations. We show that, while DMBRL algorithms perform well in OpenAI Gym, their performance can drop significantly in DeepMind Control Suite (DMC), even though these settings offer similar tasks and identical physics backends. Modern techniques designed to address several key issues that arise in these settings do not provide a consistent improvement across all environments, and overall our results show that adding synthetic rollouts to the training process -- the backbone of Dyna-style algorithms -- significantly degrades performance across most DMC environments. Our findings contribute to a deeper understanding of several fundamental challenges in model-based RL and show that, like many optimization fields, there is no free lunch when evaluating performance across diverse benchmarks in RL.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 55 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

35 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024When to Trust Your Data: Enhancing Dyna-Style Model-Based Reinforcement Learning With Data Filter16 Oct
  2. 2024Can we hop in general? A discussion of benchmark selection and design using the Hopper environment11 Oct
  3. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergence9 Mar

Other references · 32

  1. 2016Layer Normalizationcited by 5,669
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2020Reinforcement Learning with Augmented Datacited by 468
  11. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
20 more
  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  4. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  5. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  6. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  7. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  8. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  9. 2022A Survey on Model-based Reinforcement Learningcited by 66
  10. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  11. 2020Learning Off-Policy with Online Planningcited by 53
  12. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  13. 2016Learning to Perform Physics Experiments via Deep Reinforcement Learningcited by 47
  14. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  15. 2020Bidirectional Model-based Policy Optimizationcited by 38
  16. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  17. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  18. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  19. 2018The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spacescited by 15
  20. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0
  2. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  3. 2026Soft MPCritic: Value-Based Reinforcement Learning for Non-Differentiable Predictive Controlcited by 0
  4. 2026Physics-enhanced reinforcement learning for real-time optimal control of dynamical systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library
  2. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2025Scaling Offline RL via Efficient and Expressive Shortcut Models
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning
  6. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  7. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning
  8. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learning