Data to 14 Oct 2026

Compositional Reinforcement Learning from Logical Specifications

arXiv 2106.13906 ↗First posted 25 Jun 2021cs.LG cs.AIcited by 67

Not on the map. Why?

Cited by 29 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of learning control policies for complex tasks given by logical specifications. Recent approaches automatically generate a reward function from a given specification and use a suitable reinforcement learning algorithm to learn a policy that maximizes the expected reward. These approaches, however, scale poorly to complex tasks that require high-level planning. In this work, we develop a compositional learning approach, called DiRL, that interleaves high-level planning and reinforcement learning. First, DiRL encodes the specification as an abstract graph; intuitively, vertices and edges of the graph correspond to regions of the state space and simpler sub-tasks, respectively. Our approach then incorporates reinforcement learning to learn neural network policies for each edge (sub-task) within a Dijkstra-style planning algorithm to compute a high-level plan in the graph. An evaluation of the proposed approach on a set of challenging control benchmarks with continuous state and action spaces demonstrates that it outperforms state-of-the-art baselines.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 29 of its 51 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

29 of 51 references.

References · 29

2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2016OpenAI Gymcited by 1,736
2016Concrete Problems in AI Safetycited by 1,642
2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
2015End-to-End Training of Deep Visuomotor Policiescited by 875
2018Learning Dexterous In-Hand Manipulationcited by 687
2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
2018Programmatically Interpretable Reinforcement Learningcited by 193
2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
2016Reinforcement Learning With Temporal Logic Rewardscited by 117
2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77

17 more

2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 75
2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
2018Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specificationscited by 72
2020A Composable Specification Language for Reinforcement Learning Taskscited by 69
2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 67
2017Environment-Independent Task Specifications via GLTLcited by 67
2018Logically-Constrained Reinforcement Learningcited by 51
2020Neurosymbolic Reinforcement Learning with Formally Verified Explorationcited by 49
2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
2015Safety-Constrained Reinforcement Learning for MDPscited by 33
2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
2017Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learningcited by 27
2021Multi-Agent Reinforcement Learning with Temporal Logic Specificationscited by 27
2019Modular Deep Reinforcement Learning with Temporal Logic Specificationscited by 25
2021Neurosymbolic Transformers for Multi-Agent Communicationcited by 13
2020Abstract Value Iteration for Hierarchical Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 15; 2023: 13; 2024: 16; 2025: 15; 2026: 7.

67 papers.

Sort:

By area:

2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RLcited by 21
2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learningcited by 19
2023Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guaranteescited by 18
2023Policy Synthesis and Reinforcement Learning for Discounted LTLcited by 14
2023Compositional Policy Learning in Stochastic Control Systems with Formal Guaranteescited by 14
2024LTL-Constrained Policy Optimization with Cycle Experience Replaycited by 11
2024Reward Machines for Deep RL in Noisy and Uncertain Environmentscited by 11
2022Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machinescited by 10
2022Accelerated Reinforcement Learning for Temporal Logic Control Objectivescited by 9
2022Skill Machines: Temporal Logic Skill Composition in Reinforcement Learningcited by 9

38 more

2022Specification-Guided Learning of Nash Equilibria with High Social Welfarecited by 8
2023Rule-Based Error Detection and Correction to Operationalize Movement Trajectory Classificationcited by 8
2021Hierarchical Potential-based Reward Shaping from Task Specificationscited by 7
2022Learning Task Automata for Reinforcement Learning using Hidden Markov Modelscited by 7
2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
2024Directed Exploration in Reinforcement Learning from Linear Temporal Logiccited by 7
2022Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectivescited by 6
2022E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidancecited by 6
2024Adaptive Reward Design for Reinforcement Learningcited by 6
2025Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learningcited by 6
2022Categorical semantics of compositional reinforcement learningcited by 5
2023Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Surveycited by 5
2024Sample-Efficient Reinforcement Learning with Temporal Logic Objectives: Leveraging the Task Specification to Guide Explorationcited by 5
2023Assume-Guarantee Reinforcement Learningcited by 4
2024Planning with a Learned Policy Basis to Optimally Solve Complex Taskscited by 4
2022DistSPECTRL: Distributing Specifications in Multi-Agent Reinforcement Learning Systemscited by 3
2023Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specificationscited by 3
2024Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agentscited by 3
2024Inductive Generalization in Reinforcement Learning from Specificationscited by 3
2025SLAP: Shortcut Learning for Abstract Planningcited by 3
2022Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templatescited by 2
2023Automaton-Guided Curriculum Generation for Reinforcement Learning Agentscited by 2
2024Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learningcited by 2
2025HypRL: Reinforcement Learning of Control Policies for Hyperpropertiescited by 2
2025Qualitative Analysis of ω-Regular Objectives on Robust MDPscited by 2
2025Automaton Constrained Q-Learningcited by 2
2026PlatoLTL: Scaling LTL-Guided Multi-Task RLcited by 2
2022Automata Cascades: Expressivity and Sample Complexitycited by 1
2022Generalizing LTL Instructions via Future Dependent Optionscited by 1
2023Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learningcited by 1
2024Numeric Reward Machinescited by 1
2024Generalization of Compositional Tasks with Logical Specification via Implicit Planningcited by 1
2025Risk-Sensitive Agent Compositionscited by 1
2025Automata-Conditioned Cooperative Multi-Agent Reinforcement Learningcited by 1
2026Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 1
2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
2022Safe Policy Improvement in Constrained Markov Decision Processescited by 0
2023Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effectscited by 0

50 most cited of 67 All 67 (JSON)