mlsanityData to 7 Oct 2026

Composing Reinforcement Learning Policies, with Formal Guarantees

arXiv 2402.13785 ↗First posted 21 Feb 2024cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We propose a novel framework to controller design in environments with a two-level structure: a known high-level graph ("map") in which each vertex is populated by a Markov decision process, called a "room". The framework "separates concerns" by using different design techniques for low- and high-level tasks. We apply reactive synthesis for high-level tasks: given a specification as a logical formula over the high-level graph and a collection of low-level policies obtained together with "concise" latent structures, we construct a "planner" that selects which low-level policy to apply in each room. We develop a reinforcement learning procedure to train low-level policies on latent structures, which unlike previous approaches, circumvents a model distillation step. We pair the policy with probably approximately correct guarantees on its performance and on the abstraction quality, and lift these guarantees to the high-level task. These formal guarantees are the main advantage of the framework. Other advantages include scalability (rooms are large and their dynamics are unknown) and reusability of low-level policies. We demonstrate feasibility in challenging case studies where an agent navigates environments with moving obstacles and visual inputs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 74 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

33 of 74 references.

References · 33

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2017Safe Reinforcement Learning via Shieldingcited by 427
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  9. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  10. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  11. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  12. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
21 more
  1. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  2. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  3. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  4. 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 53
  5. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  6. 2020Cautious Reinforcement Learning with Logical Constraintscited by 44
  7. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 41
  8. 2021Towards Scalable Verification of Deep Reinforcement Learningcited by 33
  9. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  10. 2022Learning Control Policies for Stochastic Systems with Reach-avoid Guaranteescited by 21
  11. 2023Robust Control for Dynamical Systems With Non-Gaussian Noise via Formal Abstractionscited by 16
  12. 2021Shielding Atari Games with Bounded Presciencecited by 14
  13. 2020Abstract Value Iteration for Hierarchical Reinforcement Learningcited by 12
  14. 2023Compositional Policy Learning in Stochastic Control Systems with Formal Guaranteescited by 12
  15. 2020Formal Methods with a Touch of Magiccited by 10
  16. 2022Correct-by-Construction Runtime Enforcement in AI -- A Surveycited by 10
  17. 2019Simple Strategies in Multi-Objective MDPs (Technical Report)cited by 7
  18. 2017Deep Abstract Q-Networkscited by 6
  19. 2022Abstraction-Refinement for Hierarchical Probabilistic Modelscited by 5
  20. 2023Context-triggered Abstraction-based Control Designcited by 4
  21. 2024Pareto Curves for Compositionally Model Checking String Diagrams of MDPscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  2. 2026Value Functions as Supermartingale Certificatescited by 0