# Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning

arXiv: https://arxiv.org/abs/2502.03752 · HTML: https://mlsanity.com/paper/2502.03752/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 57 of its 77 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 57)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Demonstration-Guided Reinforcement Learning with Learned Skills](https://mlsanity.com/paper/2107.10253/) (2021)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Skill-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2207.07560/) (2022)
- [Controllability-Aware Unsupervised Skill Discovery](https://mlsanity.com/paper/2302.05103/) (2023)
- [Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards](https://mlsanity.com/paper/1910.04450/) (2019)
- [Unsupervised Meta-Learning for Reinforcement Learning](https://mlsanity.com/paper/1806.04640/) (2018)
- [Learning more skills through optimistic exploration](https://mlsanity.com/paper/2107.14226/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2204.11828/) (2022)

## Built on by (most cited first, 5 of 5)

- [Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.17062/) (2026)
- [Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2506.21039/) (2025)
- [Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.18024/) (2026)
- [Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learning](https://mlsanity.com/paper/2605.13054/) (2026)
- [Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning](https://mlsanity.com/paper/2606.20002/) (2026)

## Abstract (arXiv)

Meta-reinforcement learning (Meta-RL) facilitates rapid adaptation to unseen tasks but faces challenges in long-horizon environments. Skill-based approaches tackle this by decomposing state-action sequences into reusable skills and employing hierarchical decision-making. However, these methods are highly susceptible to noisy offline demonstrations, leading to unstable skill learning and degraded performance. To address this, we propose Self-Improving Skill Learning (SISL), which performs self-guided skill refinement using decoupled high-level and skill improvement policies, while applying skill prioritization via maximum return relabeling to focus updates on task-relevant trajectories, resulting in robust and stable adaptation even under noisy and suboptimal data. By mitigating the effect of noise, SISL achieves reliable skill learning and consistently outperforms other skill-based meta-RL methods on diverse long-horizon tasks. Our code is available at https://epsilog.github.io/SISL.