# Planning with a Learned Policy Basis to Optimally Solve Complex Tasks

arXiv: https://arxiv.org/abs/2403.15301 · HTML: https://mlsanity.com/paper/2403.15301/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 11 of 11)

- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [A Boolean Task Algebra for Reinforcement Learning](https://mlsanity.com/paper/2001.01394/) (2020)
- [Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas](https://mlsanity.com/paper/2006.01110/) (2020)
- [The Logical Options Framework](https://mlsanity.com/paper/2102.12571/) (2021)
- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning](https://mlsanity.com/paper/2006.08767/) (2020)
- [Discovering a set of policies for the worst case reward](https://mlsanity.com/paper/2102.04323/) (2021)

## Built on by (most cited first, 4 of 4)

- [Constructing an Optimal Behavior Basis for the Option Keyboard](https://mlsanity.com/paper/2505.00787/) (2025)
- [The Terminal Representation in Reinforcement Learning](https://mlsanity.com/paper/2605.31289/) (2026)
- [A Goal-Set Characterization of Task Composition in the Boolean Task Algebra](https://mlsanity.com/paper/2606.04053/) (2026)
- [Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification](https://mlsanity.com/paper/2607.29294/) (2026)

## Abstract (arXiv)

Conventional reinforcement learning (RL) methods can successfully solve a wide range of sequential decision problems. However, learning policies that can generalize predictably across multiple tasks in a setting with non-Markovian reward specifications is a challenging problem. We propose to use successor features to learn a policy basis so that each (sub)policy in it solves a well-defined subproblem. In a task described by a finite state automaton (FSA) that involves the same set of subproblems, the combination of these (sub)policies can then be used to generate an optimal solution without additional learning. In contrast to other methods that combine (sub)policies via planning, our method asymptotically attains global optimality, even in stochastic environments.