mlsanityData to 7 Oct 2026

Goal-Conditioned Data Augmentation for Offline Reinforcement Learning

arXiv 2412.20519 ↗First posted 29 Dec 2024cs.LG cs.AI cs.ROcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) enables policy learning from pre-collected offline datasets, relaxing the need to interact directly with the environment. However, limited by the quality of offline datasets, it generally fails to learn well-qualified policies in suboptimal datasets. To address datasets with insufficient optimal demonstrations, we introduce Goal-cOnditioned Data Augmentation (GODA), a novel goal-conditioned diffusion-based method for augmenting samples with higher quality. Leveraging recent advancements in generative modelling, GODA incorporates a novel return-oriented goal condition with various selection mechanisms. Specifically, we introduce a controllable scaling technique to provide enhanced return-based guidance during data sampling. GODA learns a comprehensive distribution representation of the original offline datasets while generating new data with selectively higher-return goals, thereby maximizing the utility of limited optimal demonstrations. Furthermore, we propose a novel adaptive gated conditioning method for processing noisy inputs and conditions, enhancing the capture of goal-oriented guidance. We conduct experiments on the D4RL benchmark and real-world challenges, specifically traffic signal control (TSC) tasks, to demonstrate GODA's effectiveness in enhancing data quality and superior performance compared to state-of-the-art data augmentation methods across various offline RL algorithms.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 47 references that are in the map. First posted on arXiv 29 Dec 2024.

Builds on

39 of 47 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  2. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching4 Feb
  3. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning27 Oct
  4. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning10 Apr
  5. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  6. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
14 more
  1. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  4. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  5. 2019CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenariocited by 114
  6. 2023Synthetic Experience Replaycited by 100
  7. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  8. 2024Simple Hierarchical Planning with Diffusioncited by 71
  9. 2023Elastic Decision Transformercited by 66
  10. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  11. 2021Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal controlcited by 28
  12. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  13. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 21
  14. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Multi-agent Coordination via Flow Matchingcited by 9
  2. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Modelcited by 1
  3. 2025Reachability Weighted Offline Goal-conditioned Resamplingcited by 0
  4. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
  2. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learning
  3. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  4. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
  5. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  6. 2024MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning
  7. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach
  8. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning