mlsanityData to 7 Oct 2026

Model-Based Reinforcement Learning with Multi-Task Offline Pretraining

arXiv 2306.03360 ↗First posted 6 Jun 2023cs.LG cs.AI cs.ROcited by 0

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Pretraining reinforcement learning (RL) models on offline datasets is a promising way to improve their training efficiency in online tasks, but challenging due to the inherent mismatch in dynamics and behaviors across various tasks. We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance for both dynamics representation transfer and policy transfer. We build a time-varying, domain-selective distillation loss to generate a set of offline-to-online similarity weights. These weights serve two purposes: (i) adaptively transferring the task-agnostic knowledge of physical dynamics to facilitate world model training, and (ii) learning to replay relevant source actions to guide the target policy. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

33 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Mastering Diverse Domains through World Models10 Jan
  2. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models27 May
  3. 2021Mastering Atari Games with Limited Data30 Oct
  4. 2020Mastering Atari with Discrete World Models5 Oct
  5. 2020Planning to Explore via Self-Supervised World Models12 May
  6. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 27

  1. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2018DeepMind Control Suitecited by 935
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  12. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
15 more
  1. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  2. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  3. 2019Unsupervised State Representation Learning in Ataricited by 144
  4. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  5. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  7. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  8. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  9. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  10. 2018Learning Actionable Representations from Visual Observationscited by 35
  11. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  12. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  13. 2020Unsupervised Transfer Learning for Spatiotemporal Predictive Networkscited by 9
  14. 2022S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learningcited by 9
  15. 2022Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomizationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  2. 2023Finetuning Offline World Models in the Real World
  3. 2022Can Wikipedia Help Offline Reinforcement Learning?
  4. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  5. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
  6. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  7. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  8. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models