# Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates

arXiv: https://arxiv.org/abs/2202.12967 · HTML: https://mlsanity.com/paper/2202.12967/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 22 of 22)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Deep Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1511.04143/) (2015)
- [Multi-expert learning of adaptive legged locomotion](https://mlsanity.com/paper/2012.05810/) (2020)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Efficient Bimanual Manipulation Using Learned Task Schemas](https://mlsanity.com/paper/1909.13874/) (2019)
- [Learning Multi-Level Hierarchies with Hindsight](https://mlsanity.com/paper/1712.00948/) (2017)
- [Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks](https://mlsanity.com/paper/2102.04022/) (2021)
- [Abstract Value Iteration for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2010.15638/) (2020)
- [Curiosity-Driven Multi-Criteria Hindsight Experience Replay](https://mlsanity.com/paper/1906.03710/) (2019)
- [Program Synthesis Guided Reinforcement Learning for Partially Observed Environments](https://mlsanity.com/paper/2102.11137/) (2021)
- [Planning with Abstract Learned Models While Learning Transferable Subtasks](https://mlsanity.com/paper/1912.07544/) (2019)

## Built on by (most cited first, 2 of 2)

- [REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environments](https://mlsanity.com/paper/2412.04759/) (2024)
- [Memory-Consistent Neural Networks for Imitation Learning](https://mlsanity.com/paper/2310.06171/) (2023)

## Abstract (arXiv)

Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms. A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions. We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider option templates, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it. We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude. Videos of trained agents and our code can be found at: https://sites.google.com/view/stickymittens