# Skill-Enhanced Reinforcement Learning Acceleration from Heterogeneous Demonstrations

arXiv: https://arxiv.org/abs/2412.06207 · HTML: https://mlsanity.com/paper/2412.06207/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [Demonstration-Guided Reinforcement Learning with Learned Skills](https://mlsanity.com/paper/2107.10253/) (2021)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Skill-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2207.07560/) (2022)
- [Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization](https://mlsanity.com/paper/2111.07999/) (2021)
- [Positive-Unlabeled Reward Learning](https://mlsanity.com/paper/1911.00459/) (2019)
- [Hierarchical Few-Shot Imitation with Skill Transition Models](https://mlsanity.com/paper/2107.08981/) (2021)
- [Loss factorization, weakly supervised learning and label noise robustness](https://mlsanity.com/paper/1602.02450/) (2016)
- [Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts](https://mlsanity.com/paper/2403.06966/) (2024)
- [ASPiRe:Adaptive Skill Priors for Reinforcement Learning](https://mlsanity.com/paper/2209.15205/) (2022)
- [Building Classifiers to Predict the Start of Glucose-Lowering Pharmacotherapy Using Belgian Health Expenditure Data](https://mlsanity.com/paper/1504.07389/) (2015)

## Built on by (most cited first, 1 of 1)

- [Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance](https://mlsanity.com/paper/2606.22350/) (2026)

## Abstract (arXiv)

Learning from Demonstration (LfD) is a well-established problem in Reinforcement Learning (RL), which aims to facilitate rapid RL by leveraging expert demonstrations to pre-train the RL agent. However, the limited availability of expert demonstration data often hinders its ability to effectively aid downstream RL learning. To address this problem, we propose a novel two-stage method dubbed as Skill-enhanced Reinforcement Learning Acceleration (SeRLA). SeRLA introduces a skill-level adversarial Positive-Unlabeled (PU) learning model that extracts useful skill prior knowledge by learning from both expert demonstrations and general low-cost demonstrations in the offline prior learning stage. Building on this, it employs a skill-based soft actor-critic algorithm to leverage the acquired priors for efficient training of a skill policy network in the downstream online RL stage. In addition, we propose a simple skill-level data enhancement technique to mitigate data sparsity and further improve both skill prior learning and skill policy training. Experiments across multiple standard RL benchmarks demonstrate that SeRLA achieves state-of-the-art performance in accelerating reinforcement learning on downstream tasks, particularly in the early training phase.