Data to 7 Oct 2026

How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression

arXiv 2206.03023 ↗First posted 7 Jun 2022cs.LG cs.AIcited by 65

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline goal-conditioned reinforcement learning (GCRL) promises general-purpose skill learning in the form of reaching diverse goals from purely offline datasets. We propose Goal-conditioned f-Advantage Regression (GoFAR), a novel regression-based offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a state-occupancy matching problem between a dynamics-abiding imitator agent and an expert agent that directly teleports to the goal. In contrast to prior approaches, GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks. These distinct features confer GoFAR with much better offline performance and stability as well as statistical performance guarantee that is unattainable for prior methods. Furthermore, we demonstrate that GoFAR's training objectives can be re-purposed to learn an agent-independent goal-conditioned planner from purely offline source-domain data, which enables zero-shot transfer to new target domains. Through extensive experiments, we validate GoFAR's effectiveness in various problem settings and tasks, significantly outperforming prior state-of-art. Notably, on a real robotic dexterous manipulation task, while no other method makes meaningful progress, GoFAR acquires complex manipulation behavior that successfully accomplishes diverse goals.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 59 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

39 of 59 references.

Closest ancestors · 1

Same direction, newest first

2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 38

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2022Generative Adversarial Networkscited by 5,603
2015Continuous control with deep reinforcement learningcited by 3,279
2021Convex optimizationcited by 2,479
2017Improved Training of Wasserstein GANscited by 2,223
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2016Generative Adversarial Imitation Learningcited by 1,576
2017Hindsight Experience Replaycited by 1,096
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2019Learning Latent Plans from Playcited by 310

26 more

2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
2019Efficient Exploration via State Marginal Matchingcited by 186
2018Generative Adversarial Imitation from Observationcited by 183
2019Planning with Goal-Conditioned Policiescited by 144
2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
2019Imitation Learning as f-Divergence Minimizationcited by 121
2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
2020Error Bounds of Imitating Policies and Environmentscited by 90
2021Conservative Offline Distributional Reinforcement Learningcited by 78
2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
2021Off-Policy Imitation Learning from Observationscited by 70
2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 13
2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
2021DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policiescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 16; 2024: 17; 2025: 21; 2026: 8.

65 papers.

Sort:

By area:

2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
2024Foundation Policies with Hilbert Representationscited by 89
2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25

38 more

2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
2025Horizon Generalization in Reinforcement Learningcited by 15
2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
2025Dual Goal Representationscited by 13
2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 12
2024Unsupervised-to-Online Reinforcement Learningcited by 11
2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10
2023Learning to Reach Goals via Diffusioncited by 10
2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
2025Flattening Hierarchies with Policy Bootstrappingcited by 8
2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrabilitycited by 5
2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RLcited by 5
2024Bellman Diffusion Modelscited by 4
2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
2025Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learningcited by 4
2025Hyper-GoalNet: Goal-Conditioned Manipulation Policy Learning with HyperNetworkscited by 4
2023Toward Open-ended Embodied Tasks Solvingcited by 3
2024Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spacescited by 3
2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learningcited by 3
2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learningcited by 3
2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learningcited by 2
2025Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulationscited by 2
2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 2
2024How to Solve Contextual Goal-Oriented Problems with Offline Datasets?cited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way