# When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions

arXiv: https://arxiv.org/abs/2406.07897 · HTML: https://mlsanity.com/paper/2406.07897/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 13 of 13)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPs](https://mlsanity.com/paper/1905.03814/) (2019)
- [Write, Execute, Assess: Program Synthesis with a REPL](https://mlsanity.com/paper/1906.04604/) (2019)
- [Reinforcement Learning of Theorem Proving](https://mlsanity.com/paper/1805.07563/) (2018)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Contrastive Reinforcement Learning of Symbolic Reasoning Domains](https://mlsanity.com/paper/2106.09146/) (2021)
- [LEMMA: Bootstrapping High-Level Mathematical Reasoning with Learned Symbolic Abstractions](https://mlsanity.com/paper/2211.08671/) (2022)
- [Efficient Planning under Uncertainty with Macro-actions](https://mlsanity.com/paper/1401.3827/) (2014)
- [Hardness in Markov Decision Processes: Theory and Practice](https://mlsanity.com/paper/2210.13075/) (2022)

## Built on by (most cited first, 2 of 2)

- [Offline Reinforcement Learning with Discrete Diffusion Skills](https://mlsanity.com/paper/2503.20176/) (2025)
- [Learning Multi-Timescale Interventions under Safety and Resource Constraints](https://mlsanity.com/paper/2508.03875/) (2025)

## Abstract (arXiv)

Skills are temporal abstractions that are intended to improve reinforcement learning (RL) performance through hierarchical RL. Despite our intuition about the properties of an environment that make skills useful, a precise characterization has been absent. We provide the first such characterization, focusing on the utility of deterministic skills in deterministic sparse-reward environments with finite action spaces. We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible. Additional theoretical results suggest that skills benefit exploration more than they benefit learning from existing experience, and that using unexpressive skills such as macroactions may worsen RL performance. We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.