mlsanityData to 7 Oct 2026

Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning

arXiv 2412.14865 ↗First posted 19 Dec 2024cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

We consider a Continual Reinforcement Learning setup, where a learning agent must continuously adapt to new tasks while retaining previously acquired skill sets, with a focus on the challenge of avoiding forgetting past gathered knowledge and ensuring scalability with the growing number of tasks. Such issues prevail in autonomous robotics and video game simulations, notably for navigation tasks prone to topological or kinematic changes. To address these issues, we introduce HiSPO, a novel hierarchical framework designed specifically for continual learning in navigation settings from offline data. Our method leverages distinct policy subspaces of neural networks to enable flexible and efficient adaptation to new tasks while preserving existing knowledge. We demonstrate, through a careful experimental study, the effectiveness of our method in both classical MuJoCo maze environments and complex video game-like navigation simulations, showcasing competitive performances and satisfying adaptability with respect to classical continual learning metrics, in particular regarding the memory usage and efficiency.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 48 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

39 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay16 Apr

Other references · 38

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2016Progressive Neural Networkscited by 1,332
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2018Experience Replay for Continual Learningcited by 1,074
  10. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  11. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  12. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
26 more
  1. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  2. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  3. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  4. 2018Selective Experience Replay for Lifelong Learningcited by 239
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  6. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  7. 2019Goal-conditioned Imitation Learningcited by 159
  8. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  9. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  10. 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 113
  11. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  12. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  13. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 87
  14. 2021Learning Neural Network Subspacescited by 82
  15. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  16. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  17. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
  18. 2021Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworkscited by 56
  19. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 41
  20. 2018Unsupervised Meta-Learning for Reinforcement Learningcited by 38
  21. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
  22. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  23. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14
  24. 2021Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RLcited by 8
  25. 2021Provable Hierarchy-Based Meta-Reinforcement Learningcited by 5
  26. 2023Low-Rank Multitask Learning based on Tensorized SVMs and LSSVMscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5
  2. 2025A Continual Offline Reinforcement Learning Benchmark for Navigation Taskscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  2. 2024Continual Task Learning through Adaptive Policy Self-Composition
  3. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal
  4. 2025Knowledge Retention for Continual Model-Based Reinforcement Learning
  5. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning
  6. 2019Continual Learning Using World Models for Pseudo-Rehearsal
  7. 2023Continual Visual Reinforcement Learning with A Life-Long World Model
  8. 2022The Effectiveness of World Models for Continual Reinforcement Learning