mlsanityData to 7 Oct 2026

Investigating the role of model-based learning in exploration and transfer

arXiv 2302.04009 ↗First posted 8 Feb 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

State of the art reinforcement learning has enabled training agents on tasks of ever increasing complexity. However, the current paradigm tends to favor training agents from scratch on every new task or on collections of tasks with a view towards generalizing to novel task configurations. The former suffers from poor data efficiency while the latter is difficult when test tasks are out-of-distribution. Agents that can effectively transfer their knowledge about the world pose a potential solution to these issues. In this paper, we investigate transfer learning in the context of model-based agents. Specifically, we aim to understand when exactly environment models have an advantage and why. We find that a model-based approach outperforms controlled model-free baselines for transfer learning. Through ablations, we show that both the policy and dynamics model learnt through exploration matter for successful transfer. We demonstrate our results across three domains which vary in their requirements for transfer: in-distribution procedural (Crafter), in-distribution identical (RoboDesk), and out-of-distribution (Meta-World). Our results show that intrinsic exploration combined with environment models present a viable direction towards agents that are self-supervised and able to generalize to novel reward functions.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 45 of its 65 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

45 of 65 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
33 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2016Deep Exploration via Bootstrapped DQNcited by 518
  3. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  4. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  5. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  8. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  9. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  10. 2019Self-Supervised Exploration via Disagreementcited by 263
  11. 2016Variational Intrinsic Controlcited by 262
  12. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  13. 2017Count-Based Exploration with Neural Density Modelscited by 255
  14. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  15. 2021Mastering Atari Games with Limited Datacited by 235
  16. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  17. 2021Reinforcement Learning with Prototypical Representationscited by 179
  18. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  19. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  20. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  21. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  22. 2021Learning and Planning in Complex Action Spacescited by 76
  23. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  24. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 55
  25. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  26. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  27. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  28. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  29. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  30. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  31. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  32. 2022Learning to Generalize with Object-centric Agents in the Open World Survival Game Craftercited by 13
  33. 2022CUP: Critic-Guided Policy Reusecited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 2.

5 papers.

  1. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  2. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
  3. 2023Predictive auxiliary objectives in deep RL mimic learning in the braincited by 7
  4. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  5. 2025Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learningcited by 0