mlsanityData to 7 Oct 2026

When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic

arXiv 2609.05508 ↗First posted 30 Aug 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Option-critic learns options: sub-policies together with a learned rule for when each one hands control back. Its headline result is that performance improves as options are added. We explain that result, with theory and experiment. First, the termination rule option-critic learns by maximising return contributes nothing. When the termination test and the policy that picks options read the same values, the test fires at every step, so the learned rule is identical to always terminating. When that policy explores and the test does not, as in option-critic itself, the rule can block the exploration; there are instances where it suffers Omega(T) regret while always terminating holds to O(log T). Forcing termination at every step leaves the option-count curve intact. Second, the policy inside an option barely explores at all, so a state locks onto the first action that looked good and never updates again. We name this policy necrosis, give a state-level test for it, and find three fifths of states necrotic in a typical option. Restoring exploration repairs those states, and one option then solves the task. Third, extra options improve no option; what falls is the chance that all of them fail in the same state, from 59% to 4%, and performance follows that joint quantity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

25 of 40 references.

References · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  9. 2016The Option-Critic Architecturecited by 433
  10. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  11. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  12. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
13 more
  1. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  2. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  3. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  4. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  5. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 56
  6. 2023Empirical Design in Reinforcement Learningcited by 54
  7. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  8. 2018Learning Abstract Optionscited by 31
  9. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  10. 2020Options of Interest: Temporal Abstraction with Interest Functionscited by 30
  11. 2019The Termination Criticcited by 24
  12. 2022Attention Option-Criticcited by 8
  13. 2024A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learningcited by 2

Built on by

Not cited yet.