mlsanityData to 14 Oct 2026

Synthetic Experience Replay

arXiv 2303.06614 ↗First posted 12 Mar 2023cs.LG cs.AI stat.MLcited by 101

Not on the map. Why?

Cited by 46 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SynthER), a diffusion-based approach to flexibly upsample an agent's collected experience. We show that SynthER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us to effectively train larger networks. Furthermore, SynthER enables online agents to train with a much higher update-to-data ratio than before, leading to a significant increase in sample efficiency, without any algorithmic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https://github.com/conglu1997/SynthER.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 54 of its 87 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

54 of 87 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2016OpenAI Gymcited by 1,736
  4. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  13. 2019Modeling Tabular data using Conditional GANcited by 699
  14. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  15. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  16. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  17. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  18. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  19. 2020Reinforcement Learning with Augmented Datacited by 468
  20. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 377
  21. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  22. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  23. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  24. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  25. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  26. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  27. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  28. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  29. 2020Revisiting Fundamentals of Experience Replaycited by 175
  30. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 172
  31. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  32. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  33. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  34. 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
  35. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 109
  36. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  37. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  38. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 40; 2025: 38; 2026: 15.

101 papers.

Sort:

By area:

  1. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  2. 2025Flow Q-Learningcited by 184
  3. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
  4. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  5. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 75
  6. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
  7. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  8. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 42
  9. 2024Policy-Guided Diffusioncited by 41
  10. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  11. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 40
  12. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
38 more
  1. 2023Boosting Continuous Control with Consistency Policycited by 38
  2. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 37
  3. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 36
  4. 2024Diffusion Model Predictive Controlcited by 36
  5. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 35
  6. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  7. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  8. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  9. 2024Video Diffusion Models: A Surveycited by 30
  10. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  11. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  12. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  13. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  14. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  15. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  16. 2025Generative Models in Decision Making: A Surveycited by 15
  17. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 14
  18. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 13
  19. 2024Prioritized Generative Replaycited by 13
  20. 2024RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrationscited by 13
  21. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
  22. 2025Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewritingcited by 13
  23. 2025Scalable Offline Model-Based RL with Action Chunkscited by 13
  24. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
  25. 2024DDM-Lag : A Diffusion-based Decision-making Model for Autonomous Vehicles with Lagrangian Safety Enhancementcited by 11
  26. 2024Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Controlcited by 11
  27. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  28. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  29. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  30. 2025Multi-agent Coordination via Flow Matchingcited by 10
  31. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  32. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 9
  33. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generationcited by 9
  34. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  35. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 8
  36. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  37. 2024Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replaycited by 7
  38. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7

50 most cited of 101 All 101 (JSON)