mlsanityData to 14 Oct 2026

MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning

arXiv 2412.11410 ↗First posted 16 Dec 2024cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

also in 1 more
Abstract (arXiv)

Recently, a state-of-the-art family of algorithms, known as Goal-Conditioned Weighted Supervised Learning (GCWSL) methods, has been introduced to tackle challenges in offline goal-conditioned reinforcement learning (RL). GCWSL optimizes a lower bound of the goal-conditioned RL objective and has demonstrated outstanding performance across diverse goal-reaching tasks, providing a simple, effective, and stable solution. However, prior research has identified a critical limitation of GCWSL: the lack of trajectory stitching capabilities. To address this, goal data augmentation strategies have been proposed to enhance these methods. Nevertheless, existing techniques often struggle to sample suitable augmented goals for GCWSL effectively. In this paper, we establish unified principles for goal data augmentation, focusing on goal diversity, action optimality, and goal reachability. Based on these principles, we propose a Model-based Goal Data Augmentation (MGDA) approach, which leverages a learned dynamics model to sample more suitable augmented goals. MGDA uniquely incorporates the local Lipschitz continuity assumption within the learned model to mitigate the impact of compounding errors. Empirical results show that MGDA significantly enhances the performance of GCWSL methods on both state-based and vision-based maze datasets, surpassing previous goal data augmentation techniques in improving stitching capabilities.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 57 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

27 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Swapped goal-conditioned offline reinforcement learning17 Feb
  2. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression7 Jun
  3. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 24

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  7. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  9. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  10. 2019Benchmarking Model-Based Reinforcement Learningcited by 162
  11. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  12. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
12 more
  1. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  2. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  3. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  4. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  5. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 59
  6. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  7. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  8. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 27
  9. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  10. 2022BATS: Best Action Trajectory Stitchingcited by 10
  11. 2020Grounding Language to Autonomously-Acquired Skills via Goal Generationcited by 2
  12. 2021DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Futurecited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41

Similar, not cited

Close in wording, no citation either way

  1. 2024Q-WSL: Optimizing Goal-Conditioned RL with Weighted Supervised Learning via Dynamic Programming
  2. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learning
  3. 2025Reachability Weighted Offline Goal-conditioned Resampling
  4. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  5. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  6. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
  7. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learning
  8. 2024Are Expressive Models Truly Necessary for Offline RL?