mlsanityData to 14 Oct 2026

Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data

arXiv 2508.12356 ↗First posted 17 Aug 2025cs.CV cs.AI cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) offers a promising framework for training agents using pre-collected datasets without the need for further environment interaction. However, policies trained on offline data often struggle to generalise due to limited exposure to diverse states. The complexity of visual data introduces additional challenges such as noise, distractions, and spurious correlations, which can misguide the policy and increase the risk of overfitting if the training data is not sufficiently diverse. Indeed, this makes it challenging to leverage vision-based offline data in training robust agents that can generalize to unseen environments. To solve this problem, we propose a simple approach generating additional synthetic training data. We propose a two-step process, first augmenting the originally collected offline data to improve zero-shot generalization by introducing diversity, then using a diffusion model to generate additional data in latent space. We test our method across both continuous action spaces (Visual D4RL) and discrete action spaces (Procgen), demonstrating that it significantly improves generalization without requiring any algorithmic changes to existing model-free offline RL methods. We show that our method not only increases the diversity of the training data but also significantly reduces the generalization gap at test time while maintaining computational efficiency. We believe this approach could fuel additional progress in generating synthetic data to train more general agents in the future.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 53 references that are in the map. First posted on arXiv 17 Aug 2025.

Builds on

48 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning27 May
  2. 2023The Generalization Gap in Offline Reinforcement Learning10 Dec
  3. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  10. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
28 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  3. 2020Reinforcement Learning with Augmented Datacited by 468
  4. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  5. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  7. 2020The NetHack Learning Environmentcited by 198
  8. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  10. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  11. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  12. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  13. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  14. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  15. 2019Observational Overfitting in Reinforcement Learningcited by 91
  16. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  17. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  18. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 81
  19. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  20. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  21. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  22. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  23. 2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
  24. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  25. 2020Instance based Generalization in Reinforcement Learningcited by 17
  26. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14
  27. 2024A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learningcited by 14
  28. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models
  3. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  4. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions
  5. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
  6. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learning
  7. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
  8. 2024Zero-Shot Generalization of Vision-Based RL Without Data Augmentation