# DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations

arXiv: https://arxiv.org/abs/1710.05421 · HTML: https://mlsanity.com/paper/1710.05421/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2017.

## Built on by (most cited first, 30 of 43)

- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning](https://mlsanity.com/paper/2010.13611/) (2020)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Learning and Retrieval from Prior Data for Skill-based Imitation Learning](https://mlsanity.com/paper/2210.11435/) (2022)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [Dichotomy of Control: Separating What You Can Control from What You Cannot](https://mlsanity.com/paper/2210.13435/) (2022)
- [TRAIL: Near-Optimal Imitation Learning with Suboptimal Data](https://mlsanity.com/paper/2110.14770/) (2021)
- [Few-Shot Bayesian Imitation Learning with Logical Program Policies](https://mlsanity.com/paper/1904.06317/) (2019)
- [Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics](https://mlsanity.com/paper/2211.02231/) (2022)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control](https://mlsanity.com/paper/2402.10450/) (2024)
- [Collect & Infer -- a fresh look at data-efficient Reinforcement Learning](https://mlsanity.com/paper/2108.10273/) (2021)
- [Learning Task Decomposition with Ordered Memory Policy Network](https://mlsanity.com/paper/2103.10972/) (2021)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks](https://mlsanity.com/paper/2109.10312/) (2021)
- [Language-guided Skill Learning with Temporal Variational Inference](https://mlsanity.com/paper/2402.16354/) (2024)
- [Provable Hierarchical Imitation Learning via EM](https://mlsanity.com/paper/2010.03133/) (2020)
- [SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation](https://mlsanity.com/paper/2504.15561/) (2025)
- [PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learning](https://mlsanity.com/paper/2403.00929/) (2024)
- [CompILE: Compositional Imitation Learning and Execution](https://mlsanity.com/paper/1812.01483/) (2018)
- [Transfering Hierarchical Structure with Dual Meta Imitation Learning](https://mlsanity.com/paper/2201.11981/) (2022)
- [Reinforcement learning with learned gadgets to tackle hard quantum problems on real hardware](https://mlsanity.com/paper/2411.00230/) (2024)
- [Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning](https://mlsanity.com/paper/2506.18847/) (2025)
- [Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL](https://mlsanity.com/paper/2602.03389/) (2026)
- [Identifying Selections for Unsupervised Subtask Discovery](https://mlsanity.com/paper/2410.21616/) (2024)
- [PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.06394/) (2023)
- [Hierarchical Imitation Learning for Stochastic Environments](https://mlsanity.com/paper/2309.14003/) (2023)

## Abstract (arXiv)

An option is a short-term skill consisting of a control policy for a specified region of the state space, and a termination condition recognizing leaving that region. In prior work, we proposed an algorithm called Deep Discovery of Options (DDO) to discover options to accelerate reinforcement learning in Atari games. This paper studies an extension to robot imitation learning, called Discovery of Deep Continuous Options (DDCO), where low-level continuous control skills parametrized by deep neural networks are learned from demonstrations. We extend DDO with: (1) a hybrid categorical-continuous distribution model to parametrize high-level policies that can invoke discrete options as well continuous control actions, and (2) a cross-validation method that relaxes DDO's requirement that users specify the number of options to be discovered. We evaluate DDCO in simulation of a 3-link robot in the vertical plane pushing a block with friction and gravity, and in two physical experiments on the da Vinci surgical robot, needle insertion where a needle is grasped and inserted into a silicone tissue phantom, and needle bin picking where needles and pins are grasped from a pile and categorized into bins. In the 3-link arm simulation, results suggest that DDCO can take 3x fewer demonstrations to achieve the same reward compared to a baseline imitation learning approach. In the needle insertion task, DDCO was successful 8/10 times compared to the next most accurate imitation learning baseline 6/10. In the surgical bin picking task, the learned policy successfully grasps a single object in 66 out of 99 attempted grasps, and in all but one case successfully recovered from failed grasps by retrying a second time.