mlsanityData to 14 Oct 2026

Constructing an Optimal Behavior Basis for the Option Keyboard

arXiv 2505.00787 ↗First posted 1 May 2025cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Multi-task reinforcement learning aims to quickly identify solutions for new tasks with minimal or no additional interaction with the environment. Generalized Policy Improvement (GPI) addresses this by combining a set of base policies to produce a new one that is at least as good -- though not necessarily optimal -- as any individual base policy. Optimality can be ensured, particularly in the linear-reward case, via techniques that compute a Convex Coverage Set (CCS). However, these are computationally expensive and do not scale to complex domains. The Option Keyboard (OK) improves upon GPI by producing policies that are at least as good -- and often better. It achieves this through a learned meta-policy that dynamically combines base policies. However, its performance critically depends on the choice of base policies. This raises a key question: is there an optimal set of base policies -- an optimal behavior basis -- that enables zero-shot identification of optimal solutions for any linear tasks? We solve this open problem by introducing a novel method that efficiently constructs such an optimal behavior basis. We show that it significantly reduces the number of base policies needed to ensure optimality in new tasks. We also prove that it is strictly more expressive than a CCS, enabling particular classes of non-linear tasks to be solved optimally. We empirically evaluate our technique in challenging domains and show that it outperforms state-of-the-art approaches, increasingly so as task complexity increases.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 47 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

26 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Learning Successor Features the Simple Way29 Oct
  2. 2024Planning with a Learned Policy Basis to Optimally Solve Complex Tasks22 Mar

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  5. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  6. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  7. 2020pymoo: Multi-objective Optimization in Pythoncited by 200
  8. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
  9. 2021Learning One Representation to Optimize All Rewardscited by 115
  10. 2018Universal Successor Features Approximatorscited by 103
  11. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
  12. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
12 more
  1. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  2. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  3. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
  4. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  5. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 48
  6. 2022Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfercited by 22
  7. 2021Discovering a set of policies for the worst case rewardcited by 21
  8. 2021Constructing a Good Behavior Basis for Transfer using Generalized Policy Updatescited by 14
  9. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 13
  10. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  11. 2023Combining Behaviors with the Successor Features Keyboardcited by 12
  12. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Laplacian Keyboard: Beyond the Linear Spancited by 1
  2. 2026It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the benefits of pixel-based hierarchical policies for task generalization
  2. 2024Efficient Multi-Policy Evaluation for Reinforcement Learning
  3. 2026GOPO: Policy Optimization using Ranked Rewards
  4. 2026Policy Improvement Reinforcement Learning
  5. 2026Reinforcement Learning with Decomposed Subtasks
  6. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  7. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning
  8. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space