mlsanityData to 14 Oct 2026

Unveiling Options with Neural Decomposition

arXiv 2410.11262 ↗First posted 15 Oct 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In reinforcement learning, agents often learn policies for specific tasks without the ability to generalize this knowledge to related tasks. This paper introduces an algorithm that attempts to address this limitation by decomposing neural networks encoding policies for Markov Decision Processes into reusable sub-policies, which are used to synthesize temporally extended actions, or options. We consider neural networks with piecewise linear activation functions, so that they can be mapped to an equivalent tree that is similar to oblique decision trees. Since each node in such a tree serves as a function of the input of the tree, each sub-tree is a sub-policy of the main policy. We turn each of these sub-policies into options by wrapping it with while-loops of varied number of iterations. Given the large number of options, we propose a selection mechanism based on minimizing the Levin loss for a uniform policy on these options. Empirical results in two grid-world domains where exploration can be difficult confirm that our method can identify useful options, thereby accelerating the learning process on similar but different tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 47 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

32 of 47 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2016Progressive Neural Networkscited by 1,361
  5. 2018Experience Replay for Continual Learningcited by 1,095
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  7. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  8. 2016The Option-Critic Architecturecited by 448
  9. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  10. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
20 more
  1. 2020Supermasks in Superpositioncited by 227
  2. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  3. 2018Variational Option Discovery Algorithmscited by 122
  4. 2018Tropical Geometry of Deep Neural Networkscited by 95
  5. 2017Meta Learning Shared Hierarchiescited by 93
  6. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  7. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  8. 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
  9. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  10. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 43
  11. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 40
  12. 2019Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitivescited by 30
  13. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
  14. 2018StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learningcited by 25
  15. 2018Single-Agent Policy Tree Search With Guaranteescited by 22
  16. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  17. 2019Oblique Decision Trees from Derivatives of ReLU Networkscited by 14
  18. 2019Multitask Soft Option Learningcited by 13
  19. 2017Learning to Navigate Cloth using Hapticscited by 4
  20. 2023Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networkscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization
  2. 2024In Search of Trees: Decision-Tree Policy Synthesis for Black-Box Systems via Search
  3. 2024Optimizing Interpretable Decision Tree Policies for Reinforcement Learning
  4. 2025Explainable Reinforcement Learning via Temporal Policy Decomposition
  5. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
  6. 2026Interpretable reinforcement learning with decision-tree pruning
  7. 2019PODNet: A Neural Network for Discovery of Plannable Options
  8. 2026Reinforcement Learning with Decomposed Subtasks