mlsanityData to 14 Oct 2026

Curious Replay for Model-based Adaptation

arXiv 2306.15934 ↗First posted 28 Jun 2023cs.LG cs.AI stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Agents must be able to adapt quickly as an environment changes. We find that existing model-based reinforcement learning agents are unable to do this well, in part because of how they use past experiences to train their world model. Here, we present Curious Replay -- a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal. Agents using Curious Replay exhibit improved performance in an exploration paradigm inspired by animal behavior and on the Crafter benchmark. DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter, achieving a mean score of 19.4 that substantially improves on the previous high score of 14.5 by DreamerV3 with uniform replay, while also maintaining similar performance on the Deepmind Control Suite. Code for Curious Replay is available at https://github.com/AutonomousAgentsLab/curiousreplay

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 74 of its 117 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

74 of 117 references.

References · 74

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2016Learning without Forgettingcited by 2,496
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  9. 2023Mastering Diverse Domains through World Modelscited by 1,253
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  11. 2018Experience Replay for Continual Learningcited by 1,095
  12. 2020Mastering Atari with Discrete World Modelscited by 1,007
38 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2018DeepMind Control Suitecited by 956
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  4. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  8. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  9. 2019Model-Based Reinforcement Learning for Ataricited by 502
  10. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  12. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  13. 2022Transformers are Sample-Efficient World Modelscited by 328
  14. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  15. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  16. 2018Distributed Prioritized Experience Replaycited by 269
  17. 2016Variational Intrinsic Controlcited by 267
  18. 2019Self-Supervised Exploration via Disagreementcited by 265
  19. 2014Robots that can adapt like animalscited by 204
  20. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  21. 2019Meta-Learning Representations for Continual Learningcited by 189
  22. 2021Reinforcement Learning with Prototypical Representationscited by 179
  23. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  24. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  25. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  26. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  27. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  28. 2019Online Meta-Learningcited by 137
  29. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  30. 2021Discovering and Achieving Goals via World Modelscited by 131
  31. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  32. 2018Count-Based Exploration with the Successor Representationcited by 112
  33. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  34. 2022Deep Hierarchical Planning from Pixelscited by 101
  35. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  36. 2020Rethinking Experience Replay: a Bag of Tricks for Continual Learningcited by 94
  37. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 86
  38. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 3; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Efficient World Models with Context-Aware Tokenizationcited by 33
  2. 2023A Definition of Open-Ended Learning Problems for Goal-Conditioned Agentscited by 18
  3. 2023Learning by Self-Explainingcited by 17
  4. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  5. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learningcited by 11
  6. 2026Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMscited by 11
  7. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  8. 2023Novelty Detection in Reinforcement Learning with World Modelscited by 8
  9. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  10. 2025Simulus: Combining Improvements in Sample-Efficient World Model Agentscited by 6
  11. 2025DreamerV3-XP: Optimizing exploration through uncertainty estimationcited by 2
  12. 2026Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Predictioncited by 2
4 more
  1. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  2. 2024Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learningcited by 1
  3. 2026Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learningcited by 0
  4. 2026Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
  2. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  3. 2024Prioritized Generative Replay
  4. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms
  5. 2018Interactive Agent Modeling by Learning to Probe ↗
  6. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptation
  7. 2024Partial Models for Building Adaptive Model-Based Reinforcement Learning Agents
  8. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO