Data to 7 Oct 2026

Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL

arXiv 2202.04478 ↗First posted 9 Feb 2022cs.LG cs.AIcited by 89

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences. In this paper, we revisit the theoretical property of GCSL -- optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) best-advantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and offline settings. To evaluate algorithms in the offline goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 47 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

31 of 47 references.

References · 31

2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Continuous control with deep reinforcement learningcited by 3,279
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2017Hindsight Experience Replaycited by 1,096
2016End to End Learning for Self-Driving Carscited by 1,000
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2019Behavior Regularized Offline Reinforcement Learningcited by 654
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647

19 more

2020MOPO: Model-based Offline Policy Optimizationcited by 630
2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
2019Learning Latent Plans from Playcited by 310
2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
2019Goal-conditioned Imitation Learningcited by 159
2018Self-Imitation Learningcited by 135
2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
2019Exploration via Hindsight Goal Generationcited by 53
2020Generalized Hindsight for Reinforcement Learningcited by 51
2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
2019Policy Continuation with Hindsight Inverse Dynamicscited by 22
2020Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learningcited by 2
2020Evolutionary Stochastic Policy Distillationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 24; 2024: 24; 2025: 15; 2026: 18.

89 papers.

Sort:

By area:

2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36

38 more

2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyondcited by 20
2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RLcited by 20
2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
2025Horizon Generalization in Reinforcement Learningcited by 15
2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
2022Learning Robotic Navigation from Experience: Principles, Methods, and Recent Resultscited by 13
2024Supervised Fine-Tuning as Inverse Reinforcement Learningcited by 12
2025Transitive RL: Value Learning via Divide and Conquercited by 12
2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learningcited by 10
2023Learning to Reach Goals via Diffusioncited by 10
2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
2025Flattening Hierarchies with Policy Bootstrappingcited by 8
2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2022Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shapingcited by 6
2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
2023ReRoGCRL: Representation-based Robustness in Goal-Conditioned Reinforcement Learningcited by 6
2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
2022What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorizationcited by 5
2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrabilitycited by 5
2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way