mlsanityData to 7 Oct 2026

Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks

arXiv 2408.11300 ↗First posted 21 Aug 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Goal-conditioned (GC) policy learning often faces a challenge arising from the sparsity of rewards, when confronting long-horizon goals. To address the challenge, we explore skill-based GC policy learning in offline settings, where skills are acquired from existing data and long-horizon goals are decomposed into sequences of near-term goals that align with these skills. Specifically, we present an `offline GC policy learning via skill-step abstraction' framework (GLvSA) tailored for tackling long-horizon GC tasks affected by goal distribution shifts. In the framework, a GC policy is progressively learned offline in conjunction with the incremental modeling of skill-step abstractions on the data. We also devise a GC policy hierarchy that not only accelerates GC policy learning within the framework but also allows for parameter-efficient fine-tuning of the policy. Through experiments with the maze and Franka kitchen environments, we demonstrate the superiority and efficiency of our GLvSA framework in adapting GC policies to a wide range of long-horizon goals. The framework achieves competitive zero-shot and few-shot adaptation performance, outperforming existing GC policy learning and skill-based methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 32 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

27 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression7 Jun
  2. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2020Mastering Atari with Discrete World Modelscited by 990
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  7. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  8. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  9. 2021Discovering and Achieving Goals via World Modelscited by 131
  10. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  11. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  12. 2022Skill-based Model-based Reinforcement Learningcited by 50
13 more
  1. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  2. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  3. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  4. 2022Skill-based Meta-Reinforcement Learningcited by 36
  5. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  6. 2023Planning Goals for Explorationcited by 28
  7. 2022Scalable Multi-Agent Model-Based Reinforcement Learningcited by 24
  8. 2022Learning Options via Compressioncited by 22
  9. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17
  10. 2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 14
  11. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  12. 2023Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skillscited by 6
  13. 2022D-Shape: Demonstration-Shaped Reinforcement Learning via Goal Conditioningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  2. 2024Robust Policy Learning via Offline Skill Diffusion
  3. 2025Flattening Hierarchies with Policy Bootstrapping
  4. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
  5. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  6. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  7. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
  8. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learning