mlsanityData to 7 Oct 2026

When to use parametric models in reinforcement learning?

arXiv 1906.05243 ↗First posted 12 Jun 2019cs.LG cs.AI stat.MLcited by 118

Not on the map. Why?

Abstract (arXiv)

We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan: to use more computation without additional data to improve predictions and behaviour. We discuss when to expect benefits from either approach, and interpret prior work in this context. We hypothesise that, under suitable conditions, replay-based algorithms should be competitive to or better than model-based algorithms if the model is used only to generate fictional transitions from observed states for an update rule that is otherwise model-free. We validated this hypothesis on Atari 2600 video games. The replay-based algorithm attained state-of-the-art data efficiency, improving over prior results with parametric models.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 30; 2023: 22; 2024: 19; 2025: 12; 2026: 6.

118 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  4. 2021Mastering Atari Games with Limited Datacited by 235
  5. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  6. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  7. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  8. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  9. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  10. 2021APS: Active Pretraining with Successor Featurescited by 109
  11. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  12. 2021The Efficiency Misnomercited by 98
38 more
  1. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 88
  2. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  3. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  4. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  5. 2021Vector Quantized Models for Planningcited by 47
  6. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  7. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
  8. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  9. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  10. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  11. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  12. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  13. 2021Zoo-Tuning: Adaptive Transfer from a Zoo of Modelscited by 35
  14. 2023Small batch deep reinforcement learningcited by 35
  15. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  16. 2022Human-level Atari 200x fastercited by 28
  17. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  18. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  19. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  20. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  21. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  22. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  23. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  24. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 18
  25. 2022Topological Experience Replaycited by 17
  26. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  27. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  28. 2022A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learningcited by 15
  29. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  30. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  31. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  32. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  33. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  34. 2022Goal-Space Planning with Subgoal Modelscited by 12
  35. 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 12
  36. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  37. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  38. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11

50 most cited of 118 All 118 (JSON)