mlsanityData to 14 Oct 2026

OptionZero: Planning with Learned Options

arXiv 2502.16634 ↗First posted 23 Feb 2025cs.AI cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Planning with options -- a sequence of primitive actions -- has been shown effective in reinforcement learning within complex environments. Previous studies have focused on planning with predefined options or learned options through expert demonstration data. Inspired by MuZero, which learns superhuman heuristics without any human knowledge, we propose a novel approach, named OptionZero. OptionZero incorporates an option network into MuZero, providing autonomous discovery of options through self-play games. Furthermore, we modify the dynamics network to provide environment transitions when using options, allowing searching deeper under the same simulation constraints. Empirical experiments conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero, achieving a 131.58% improvement in mean human-normalized score. Our behavior analysis shows that OptionZero not only learns options but also acquires strategic skills tailored to different game characteristics. Our findings show promising directions for discovering and using options in planning. Our code is available at https://rlg.iis.sinica.edu.tw/papers/optionzero.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 39 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

25 of 39 references.

References · 25

  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2016The Option-Critic Architecturecited by 448
  6. 2021Mastering Atari Games with Limited Datacited by 237
  7. 2021Learning and Planning in Complex Action Spacescited by 77
  8. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  9. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 39
  10. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 30
  11. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 28
  12. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
13 more
  1. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  2. 2018Finding Options that Minimize Planning Timecited by 17
  3. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  4. 2016Deep Reinforcement Learning With Macro-Actionscited by 14
  5. 2023MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Gamescited by 14
  6. 2021Visualizing MuZero Modelscited by 9
  7. 2019On the Role of Weight Sharing During Deep Option Learningcited by 8
  8. 2023What model does MuZero learn?cited by 7
  9. 2023LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Frameworkcited by 7
  10. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  11. 2023Optimizing Memory Mapping Using Deep Reinforcement Learningcited by 3
  12. 2023Iterative Option Discovery for Planning, by Planningcited by 1
  13. 2024Demystifying MuZero Planning: Interpreting the Learned Modelcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 23
  2. 2025Toward Agents That Reason About Their Computationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Mastering the Game of Go with Self-play Experience Replay
  2. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
  3. 2026Mastering Atari 2600 Games with Discovered Options
  4. 2025Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations
  5. 2023Thinker: Learning to Plan and Act
  6. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Models
  7. 2025Scalable Decision Making for Games of Imperfect Information
  8. 2026Regret-Guided Search Control for Efficient Learning in AlphaZero