Data to 7 Oct 2026

What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?

arXiv 2305.18882 ↗First posted 30 May 2023cs.LG cs.AIcited by 36

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline goal-conditioned RL (GCRL) offers a way to train general-purpose agents from fully offline datasets. In addition to being conservative within the dataset, the generalization ability to achieve unseen goals is another fundamental challenge for offline GCRL. However, to the best of our knowledge, this problem has not been well studied yet. In this paper, we study out-of-distribution (OOD) generalization of offline GCRL both theoretically and empirically to identify factors that are important. In a number of experiments, we observe that weighted imitation learning enjoys better generalization than pessimism-based offline RL method. Based on this insight, we derive a theory for OOD generalization, which characterizes several important design choices. We then propose a new offline GCRL method, Generalizable Offline goAl-condiTioned RL (GOAT), by combining the findings from our theoretical and empirical studies. On a new benchmark containing 9 independent identically distributed (IID) tasks and 17 OOD tasks, GOAT outperforms current state-of-the-art methods by a large margin.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 81 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

58 of 81 references.

Closest ancestors · 2

Same direction, newest first

Other references · 56

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2019Invariant Risk Minimizationcited by 1,832
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2017Hindsight Experience Replaycited by 1,096
2018Exploration by Random Network Distillationcited by 966
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
2021Domain Generalization: A Surveycited by 760

38 more

2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2013Domain Generalization via Invariant Feature Representationcited by 622
2021Domain Generalization with MixStylecited by 563
2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2018Quantifying Generalization in Reinforcement Learningcited by 345
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
2020Learning to Generate Novel Domains for Domain Generalizationcited by 296
2019Self-Supervised Exploration via Disagreementcited by 263
2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
2022Multi-Game Decision Transformerscited by 212
2020Prioritized Level Replaycited by 190
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2018Universal Successor Features Approximatorscited by 102
2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
2020Error Bounds of Imitating Policies and Environmentscited by 90
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58
2020Suphx: Mastering Mahjong with Deep Reinforcement Learningcited by 55
2020Generalized Hindsight for Reinforcement Learningcited by 51
2023Anti-Exploration by Random Network Distillationcited by 49
2021Offline Reinforcement Learning as Anti-Explorationcited by 44
2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 7; 2026: 7.

36 papers.

Sort:

By area:

2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024Foundation Policies with Hilbert Representationscited by 89
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21

24 more

2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
2024Unsupervised-to-Online Reinforcement Learningcited by 11
2025Flattening Hierarchies with Policy Bootstrappingcited by 8
2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
2024Episodic Reinforcement Learning with Expanded State-reward Spacecited by 3
2024Preference Goal Tuning: Post-Training as Latent Control for Frozen Policiescited by 3
2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learningcited by 3
2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
2024Offline Goal-Conditioned Reinforcement Learning for Shape Control of Deformable Linear Objectscited by 1
2024How to Solve Contextual Goal-Oriented Problems with Offline Datasets?cited by 1
2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agentscited by 1
2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
2026Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamicscited by 0
2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0
2026SUN: Reaching for Novelty in Reinforcement Learningcited by 0
2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0

Similar, not cited

Close in wording, no citation either way