mlsanityData to 14 Oct 2026

Verifiable and Compositional Reinforcement Learning Systems

arXiv 2106.05864 ↗First posted 7 Jun 2021cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

We propose a framework for verifiable and compositional reinforcement learning (RL) in which a collection of RL subsystems, each of which learns to accomplish a separate subtask, are composed to achieve an overall task. The framework consists of a high-level model, represented as a parametric Markov decision process (pMDP) which is used to plan and to analyze compositions of subsystems, and of the collection of low-level subsystems themselves. By defining interfaces between the subsystems, the framework enables automatic decompositions of task specifications, e.g., reach a target set of states with a probability of at least 0.95, into individual subtask specifications, i.e. achieve the subsystem's exit conditions with at least some minimum probability, given that its entry conditions are met. This in turn allows for the independent training and testing of the subsystems; if they each learn a policy satisfying the appropriate subtask specification, then their composition is guaranteed to satisfy the overall task specification. Conversely, if the subtask specifications cannot all be satisfied by the learned policies, we present a method, formulated as the problem of finding an optimal set of parameters in the pMDP, to automatically update the subtask specifications to account for the observed shortcomings. The result is an iterative procedure for defining subtask specifications, and for training the subsystems to meet them. As an additional benefit, this procedure allows for particularly challenging or important components of an overall task to be determined automatically, and focused on, during training. Experimental results demonstrate the presented framework's novel capabilities.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 12 of its 33 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

12 of 33 references.

References · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  4. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  5. 2018Unity: A General Platform for Intelligent Agentscited by 236
  6. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  7. 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 75
  8. 2017Environment-Independent Task Specifications via GLTLcited by 67
  9. 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
  10. 2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
  11. 2020SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learningcited by 23
  12. 2018Synthesis in pMDPs: A Tale of 1001 Parameterscited by 10

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 6; 2024: 0; 2025: 4.

13 papers.

Sort:

By area:

  1. 2023Compositional Policy Learning in Stochastic Control Systems with Formal Guaranteescited by 14
  2. 2022Learning Task Automata for Reinforcement Learning using Hidden Markov Modelscited by 7
  3. 2022Ablation Study of How Run Time Assurance Impacts the Training and Performance of Reinforcement Learning Agentscited by 5
  4. 2022Categorical semantics of compositional reinforcement learningcited by 5
  5. 2023Verifiable Reinforcement Learning Systems via Compositionalitycited by 3
  6. 2023Efficient Sensitivity Analysis for Parametric Robust Markov Chainscited by 2
  7. 2025Compositional shield synthesis for safe reinforcement learning in partial observabilitycited by 2
  8. 2023Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learningcited by 1
  9. 2023Verified Compositional Neuro-Symbolic Control for Stochastic Systems with Temporal Logic Taskscited by 0
  10. 2023A Multifidelity Sim-to-Real Pipeline for Verifiable and Compositional Reinforcement Learningcited by 0
  11. 2025A Unified Theory of Compositionality, Modularity, and Interpretability in Markov Decision Processescited by 0
  12. 2025Automating the Refinement of Reinforcement Learning Specificationscited by 0
1 more
  1. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learningcited by 0