mlsanityData to 7 Oct 2026

GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning

arXiv 2405.16907 ↗First posted 27 May 2024cs.AI cs.LGcited by 22

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (Offline RL) presents challenges of learning effective decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection and data synthesizing, aim to improve Q-function approximation by smoothing the learned state-action region. However, these methods often fall short of directly improving the quality of offline datasets, leading to suboptimal results. In response, we introduce GTA, Generative Trajectory Augmentation, a novel generative data augmentation approach designed to enrich offline data by augmenting trajectories to be both high-rewarding and dynamically plausible. GTA applies a diffusion model within the data augmentation framework. GTA partially noises original trajectories and then denoises them with classifier-free guidance via conditioning on amplified return value. Our results show that GTA, as a general data augmentation strategy, enhances the performance of widely used offline RL algorithms across various tasks with unique challenges. Furthermore, we conduct a quality analysis of data augmented by GTA and demonstrate that GTA improves the quality of the data. Our code is available at https://github.com/Jaewoopudding/GTA

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 48 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

42 of 48 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  3. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  10. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
13 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  3. 2020Reinforcement Learning with Augmented Datacited by 452
  4. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  5. 2022Biological Sequence Design with GFlowNetscited by 172
  6. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  7. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  8. 2023Synthetic Experience Replaycited by 100
  9. 2019Model Inversion Networks for Model-Based Optimizationcited by 83
  10. 2023Diffusion Models for Black-Box Optimizationcited by 75
  11. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  12. 2024Policy-Guided Diffusioncited by 40
  13. 2023Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequencescited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 5.

22 papers.

Sort:

By area:

  1. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
  2. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  3. 2024Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimizationcited by 15
  4. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  5. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  6. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  7. 2025Multi-agent Coordination via Flow Matchingcited by 9
  8. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
  9. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 8
  10. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  11. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 7
  12. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
10 more
  1. 2025BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learningcited by 4
  2. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  3. 2025Offline Reinforcement Learning via Inverse Optimizationcited by 2
  4. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 2
  5. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  6. 2024Rejecting Hallucinated State Targets during Planningcited by 1
  7. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Datacited by 0
  8. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learningcited by 0
  9. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0
  10. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  2. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learning
  3. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning
  4. 2025Offline Reinforcement Learning with Generative Trajectory Policies
  5. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
  6. 2024Offline Trajectory Optimization for Offline Reinforcement Learning
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2026MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning