mlsanityData to 7 Oct 2026

Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning

arXiv 2305.15260 ↗First posted 24 May 2023cs.LGcited by 7

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Training offline RL models using visual inputs poses two significant challenges, i.e., the overfitting problem in representation learning and the overestimation bias for expected future rewards. Recent work has attempted to alleviate the overestimation bias by encouraging conservative behaviors. This paper, in contrast, tries to build more flexible constraints for value estimation without impeding the exploration of potential advantages. The key idea is to leverage off-the-shelf RL simulators, which can be easily interacted with in an online manner, as the "test bed" for offline policies. To enable effective online-to-offline knowledge transfer, we introduce CoWorld, a model-based RL approach that mitigates cross-domain discrepancies in state and reward spaces. Experimental results demonstrate the effectiveness of CoWorld, outperforming existing RL approaches by large margins.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 55 of its 68 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

55 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Continual Visual Reinforcement Learning with A Life-Long World Model12 Mar
  2. 2022Transformers are Sample-Efficient World Models1 Sep
  3. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models27 May
  4. 2020Mastering Atari with Discrete World Models5 Oct
  5. 2020Planning to Explore via Self-Supervised World Models12 May
  6. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 49

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  9. 2018DeepMind Control Suitecited by 935
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
37 more
  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  6. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  7. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  8. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  9. 2022Masked Visual Pre-training for Motor Controlcited by 246
  10. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  12. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  13. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  14. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  15. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  16. 2020Invariant Causal Prediction for Block MDPscited by 104
  17. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  18. 2022Deep Hierarchical Planning from Pixelscited by 96
  19. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  20. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  21. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  22. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  23. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  24. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  25. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  26. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  27. 2023Behavior Proximal Policy Optimizationcited by 43
  28. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  29. 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
  30. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  31. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  32. 2022Transfer RL across Observation Feature Spaces via Model-Based Regularizationcited by 18
  33. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  34. 2022S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learningcited by 9
  35. 2023Cross-domain Random Pre-training with Prototypes for Reinforcement Learningcited by 9
  36. 2020TempLe: Learning Template of Transitions for Sample Efficient Multi-task RLcited by 6
  37. 2022CtrlFormer: Learning Transferable State Representation for Visual Control via Transformercited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  2. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  3. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  4. 2026MetaMind: General and Cognitive World Models in Multi-Agent Systems by Meta-Theory of Mindcited by 3
  5. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1
  6. 2026DreamSAC: Learning Hamiltonian World Models via Symmetry Explorationcited by 1
  7. 2026ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models
  2. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  3. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  6. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order