# The Logical Options Framework

arXiv: https://arxiv.org/abs/2102.12571 · HTML: https://mlsanity.com/paper/2102.12571/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 24 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications](https://mlsanity.com/paper/1807.06333/) (2018)
- [PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning](https://mlsanity.com/paper/1710.03937/) (2017)
- [A Composable Specification Language for Reinforcement Learning Tasks](https://mlsanity.com/paper/2008.09293/) (2020)
- [Learning Actionable Representations with Goal-Conditioned Policies](https://mlsanity.com/paper/1811.07819/) (2018)
- [Logically-Constrained Reinforcement Learning](https://mlsanity.com/paper/1801.08099/) (2018)
- [PyRep: Bringing V-REP to Deep Robot Learning](https://mlsanity.com/paper/1906.11176/) (2019)
- [Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas](https://mlsanity.com/paper/2006.01110/) (2020)
- [Modular Deep Reinforcement Learning with Temporal Logic Specifications](https://mlsanity.com/paper/1909.11591/) (2019)
- [Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning](https://mlsanity.com/paper/2006.08767/) (2020)
- [Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments](https://mlsanity.com/paper/1703.07887/) (2017)
- [Planning With Uncertain Specifications (PUnS)](https://mlsanity.com/paper/1906.03218/) (2019)
- [What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning](https://mlsanity.com/paper/1901.01492/) (2019)
- [Planning with State Abstractions for Non-Markovian Task Specifications](https://mlsanity.com/paper/1905.12096/) (2019)
- [A Survey of Knowledge-based Sequential Decision Making under Uncertainty](https://mlsanity.com/paper/2008.08548/) (2020)

## Built on by (most cited first, 25 of 25)

- [DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL](https://mlsanity.com/paper/2410.04631/) (2024)
- [Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning](https://mlsanity.com/paper/2310.03718/) (2023)
- [In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications](https://mlsanity.com/paper/2110.09461/) (2021)
- [Skill Expansion and Composition in Parameter Space](https://mlsanity.com/paper/2502.05932/) (2025)
- [Skill Machines: Temporal Logic Skill Composition in Reinforcement Learning](https://mlsanity.com/paper/2205.12532/) (2022)
- [Learning Task Automata for Reinforcement Learning using Hidden Markov Models](https://mlsanity.com/paper/2208.11838/) (2022)
- [Categorical semantics of compositional reinforcement learning](https://mlsanity.com/paper/2208.13687/) (2022)
- [Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.03752/) (2025)
- [Hierarchies of Reward Machines](https://mlsanity.com/paper/2205.15752/) (2022)
- [Planning with a Learned Policy Basis to Optimally Solve Complex Tasks](https://mlsanity.com/paper/2403.15301/) (2024)
- [FORM: Learning Expressive and Transferable First-Order Logic Reward Machines](https://mlsanity.com/paper/2501.00364/) (2024)
- [Learning Reward Machines from Partially Observed Policies](https://mlsanity.com/paper/2502.03762/) (2025)
- [Hierarchical Reinforcement Learning with AI Planning Models](https://mlsanity.com/paper/2203.00669/) (2022)
- [Safety-Constrained Policy Transfer with Successor Features](https://mlsanity.com/paper/2211.05361/) (2022)
- [Automaton Constrained Q-Learning](https://mlsanity.com/paper/2510.05061/) (2025)
- [Generalizing LTL Instructions via Future Dependent Options](https://mlsanity.com/paper/2212.04576/) (2022)
- [Generalization of Compositional Tasks with Logical Specification via Implicit Planning](https://mlsanity.com/paper/2410.09686/) (2024)
- [Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systems](https://mlsanity.com/paper/2506.05577/) (2025)
- [Grounding LTL Tasks in Sub-Symbolic RL Environments for Zero-Shot Generalization](https://mlsanity.com/paper/2602.09761/) (2026)
- [Zero-Shot Instruction Following in RL via Structured LTL Representations](https://mlsanity.com/paper/2602.14344/) (2026)
- [BXRL: Behavior-Explainable Reinforcement Learning](https://mlsanity.com/paper/2603.23738/) (2026)
- [SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning](https://mlsanity.com/paper/2604.24729/) (2026)
- [Zero-Shot Instruction Following in RL via Structured LTL Representations](https://mlsanity.com/paper/2512.02633/) (2025)
- [Do It for HER: First-Order Temporal Logic Reward Specification in Reinforcement Learning (Extended Version)](https://mlsanity.com/paper/2602.06227/) (2026)
- [Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL](https://mlsanity.com/paper/2609.38065/) (2026)

## Abstract (arXiv)

Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF's learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.