mlsanityData to 7 Oct 2026

S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning

arXiv 2209.15256 ↗First posted 30 Sep 2022cs.LG cs.CV cs.ROcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (Offline RL) suffers from the innate distributional shift as it cannot interact with the physical environment during training. To alleviate such limitation, state-based offline RL leverages a learned dynamics model from the logged experience and augments the predicted state transition to extend the data distribution. For exploiting such benefit also on the image-based RL, we firstly propose a generative model, S2P (State2Pixel), which synthesizes the raw pixel of the agent from its corresponding state. It enables bridging the gap between the state and the image domain in RL algorithms, and virtually exploring unseen image distribution via model-based transition in the state space. Through experiments, we confirm that our S2P-based image synthesis not only improves the image-based offline RL performance but also shows powerful generalization capability on unseen tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 68 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

50 of 68 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  4. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2015Continuous control with deep reinforcement learningcited by 3,279
  12. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
28 more
  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  8. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  14. 2020Reinforcement Learning with Augmented Datacited by 452
  15. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  16. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  17. 2016A Learned Representation For Artistic Stylecited by 340
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  19. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  20. 2017Geometric GANcited by 232
  21. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  22. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  23. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  24. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  25. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  26. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  27. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  28. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 1; 2026: 1.

9 papers.

  1. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  2. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
  3. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  4. 2025Generative Models in Decision Making: A Surveycited by 14
  5. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  6. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  7. 2024Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensorscited by 4
  8. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretrainingcited by 0
  9. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
  2. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
  3. 2021A Minimalist Approach to Offline Reinforcement Learning
  4. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
  5. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  6. 2024State-Constrained Offline Reinforcement Learning
  7. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  8. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling