mlsanityData to 7 Oct 2026

On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning

arXiv 2210.10763 ↗First posted 19 Oct 2022cs.LG cs.CV cs.ROcited by 15

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) algorithms can solve challenging control problems directly from image observations, but they often require millions of environment interactions to do so. Recently, model-based RL algorithms have greatly improved sample-efficiency by concurrently learning an internal model of the world, and supplementing real environment interactions with imagined rollouts for policy improvement. However, learning an effective model of the world from scratch is challenging, and in stark contrast to humans that rely heavily on world understanding and visual cues for learning new skills. In this work, we investigate whether internal models learned by modern model-based RL algorithms can be leveraged to solve new, distinctly different tasks faster. We propose Model-Based Cross-Task Transfer (XTRA), a framework for sample-efficient online RL with scalable pretraining and finetuning of learned world models. By offline multi-task pretraining and online cross-task finetuning, we achieve substantial improvements over a baseline trained from scratch; we improve mean performance of model-based algorithm EfficientZero by 23%, and by as much as 71% in some instances.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 54 of its 66 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

54 of 66 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
38 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2022A Generalist Agentcited by 802
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2019Model-Based Reinforcement Learning for Ataricited by 495
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  12. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  13. 2020Reinforcement Learning with Augmented Datacited by 452
  14. 2015Policy Distillationcited by 414
  15. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  16. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  17. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  18. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  19. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  20. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  21. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  22. 2022Masked Visual Pre-training for Motor Controlcited by 246
  23. 2021Mastering Atari Games with Limited Datacited by 235
  24. 2022Online Decision Transformercited by 194
  25. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  26. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  27. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  28. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  29. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  30. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  31. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  32. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  33. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  34. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  35. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  36. 2018Investigating Human Priors for Playing Video Gamescited by 39
  37. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  38. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 4; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  3. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  4. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  5. 2023Finetuning Offline World Models in the Real Worldcited by 39
  6. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  7. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  8. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  9. 2025Neural Motion Simulator: Pushing the Limit of World Models in Reinforcement Learningcited by 11
  10. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  11. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
  12. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
3 more
  1. 2023Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Promptscited by 3
  2. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretrainingcited by 0
  3. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-Learning With World Models
  2. 2024PWM: Policy Learning with Multi-Task World Models
  3. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  4. 2026WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learning
  5. 2023HarmonyDream: Task Harmonization Inside World Models
  6. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  7. 2026WAM-RL: World-Action Model Reinforcement Learning with Reconstruction Rewards and Online Video SFT
  8. 2026Cross-Benchmark Generalization in Long-Horizon Agents