mlsanityData to 14 Oct 2026

Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach

arXiv 2505.06482 ↗First posted 10 May 2025cs.LG cs.AI cs.ROcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers commonsense knowledge of control policy and physical dynamics from natural videos to the RL agent within the target domain. VeoRL achieves substantial performance gains (over 100% in some cases) across visual control tasks in robotic manipulation, autonomous driving, and open-world video games.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 71 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

56 of 71 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning10 Jun
  2. 2024Learning from Sparse Offline Datasets via Conservative Density Estimation16 Jan
  3. 2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexity31 Oct
  4. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  5. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning24 May
  6. 2022Contrastive Value Learning: Implicit Models for Simple Offline RL3 Nov
  7. 2022Behavior Prior Representation learning for Offline Reinforcement Learning2 Nov
  8. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  9. 2022Learning Value Functions from Undirected State-only Experience26 Apr
  10. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
5 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  7. 2023Mastering Diverse Domains through World Modelscited by 1,253
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  10. 2020Mastering Atari with Discrete World Modelscited by 1,007
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
27 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  5. 2018Behavioral Cloning from Observationcited by 492
  6. 2020Reinforcement Learning with Augmented Datacited by 468
  7. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  10. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  11. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  12. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  13. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
  14. 2022Deep Hierarchical Planning from Pixelscited by 101
  15. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  16. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  17. 2022Offline Visual Representation Learning for Embodied Navigationcited by 75
  18. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  19. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 42
  20. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  21. 2022Offline Reinforcement Learning for Visual Navigationcited by 29
  22. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  23. 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26
  24. 2024Playable Game Generationcited by 26
  25. 2023Bidirectional Learning for Offline Model-based Biological Sequence Designcited by 20
  26. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  27. 2023Contrastive Example-Based Controlcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Planning with Reasoning using Vision Language World Modelcited by 36

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  3. 2020MOReL : Model-Based Offline Reinforcement Learning
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  6. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  7. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models
  8. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning