mlsanityData to 7 Oct 2026

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations

arXiv 2206.04779 ↗First posted 9 Jun 2022cs.LG cs.AI cs.CVcited by 63

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning has shown great promise in leveraging large pre-collected datasets for policy learning, allowing agents to forgo often-expensive online data collection. However, offline reinforcement learning from visual observations with continuous action spaces remains under-explored, with a limited understanding of the key challenges in this complex domain. In this paper, we establish simple baselines for continuous control in the visual domain and introduce a suite of benchmarking tasks for offline reinforcement learning from visual observations designed to better represent the data distributions present in real-world offline RL problems and guided by a set of desiderata for offline RL from visual observations, including robustness to visual distractions and visually identifiable changes in dynamics. Using this suite of benchmarking tasks, we show that simple modifications to two popular vision-based online reinforcement learning algorithms, DreamerV2 and DrQ-v2, suffice to outperform existing offline RL methods and establish competitive baselines for continuous control in the visual domain. We rigorously evaluate these algorithms and perform an empirical evaluation of the differences between state-of-the-art model-based and model-free offline RL methods for continuous control from visual observations. All code and data used in this evaluation are open-sourced to facilitate progress in this domain.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 71 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

51 of 71 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  6. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
27 more
  1. 2020Mastering Atari with Discrete World Modelscited by 990
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  6. 2020Reinforcement Learning with Augmented Datacited by 452
  7. 2021Implicit Behavioral Cloningcited by 442
  8. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  9. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  10. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  11. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2018Understanding disentangling in β-VAEcited by 332
  14. 2018Learning to Drive in a Daycited by 266
  15. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  16. 2020Model-Based Offline Planningcited by 126
  17. 2018State Representation Learning for Control: An Overviewcited by 125
  18. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  19. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  20. 2019Observational Overfitting in Reinforcement Learningcited by 90
  21. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  22. 2020ViNG: Learning Open-World Navigation with Visual Goalscited by 77
  23. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  24. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  25. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  26. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  27. 2017Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processescited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 21; 2024: 14; 2025: 13; 2026: 8.

63 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  5. 2023Synthetic Experience Replaycited by 100
  6. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  7. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  8. 2023Learning to Modulate pre-trained Models in RLcited by 28
  9. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  10. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  11. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 25
  12. 2022Joint Embedding Predictive Architectures Focus on Slow Featurescited by 24
38 more
  1. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  2. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  3. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  4. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
  5. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
  6. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  7. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  8. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  9. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  10. 2024Prioritized Generative Replaycited by 13
  11. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  12. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  13. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  14. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learningcited by 11
  15. 2023Ignorance is Bliss: Robust Control via Information Gatingcited by 10
  16. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  17. 2025A Clean Slate for Offline Reinforcement Learningcited by 10
  18. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  19. 2023Minimax Optimal Q Learning with Nearest Neighborscited by 8
  20. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learningcited by 8
  21. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  22. 2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 7
  23. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  24. 2026Hallucination in World Models is Predictable and Preventablecited by 7
  25. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  26. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
  27. 2023Action Inference by Maximising Evidence: Zero-Shot Imitation from Observation with World Modelscited by 5
  28. 2024DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractorscited by 5
  29. 2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
  30. 2023PcLast: Discovering Plannable Continuous Latent Statescited by 4
  31. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  32. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  33. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  34. 2022Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learningcited by 3
  35. 2024Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPscited by 2
  36. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  37. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologiescited by 1
  38. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning Hands-On
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  3. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models
  4. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning