# Joint Learning of Policy with Unknown Temporal Constraints for Safe Reinforcement Learning

arXiv: https://arxiv.org/abs/2305.00576 · HTML: https://mlsanity.com/paper/2305.00576/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 30 Apr 2023.

## Builds on (most cited first, 12 of 12)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications](https://mlsanity.com/paper/1705.06270/) (2017)
- [Logically-Constrained Reinforcement Learning](https://mlsanity.com/paper/1801.08099/) (2018)
- [Deep Reinforcement Learning with Enhanced Safety for Autonomous Highway Driving](https://mlsanity.com/paper/1910.12905/) (2019)
- [Interpretable Classification of Time-Series Data using Efficient Enumerative Techniques](https://mlsanity.com/paper/1907.10265/) (2019)
- [A Survey on Reinforcement Learning in Aviation Applications](https://mlsanity.com/paper/2211.02147/) (2022)
- [A Robust Genetic Algorithm for Learning Temporal Specifications from Data](https://mlsanity.com/paper/1711.06202/) (2017)
- [Differentiable Inference of Temporal Logic Formulas](https://mlsanity.com/paper/2208.05440/) (2022)

## Built on by (most cited first, 1 of 1)

- [Learning Formal Specifications from Membership and Preference Queries](https://mlsanity.com/paper/2307.10434/) (2023)

## Abstract (arXiv)

In many real-world applications, safety constraints for reinforcement learning (RL) algorithms are either unknown or not explicitly defined. We propose a framework that concurrently learns safety constraints and optimal RL policies in such environments, supported by theoretical guarantees. Our approach merges a logically-constrained RL algorithm with an evolutionary algorithm to synthesize signal temporal logic (STL) specifications. The framework is underpinned by theorems that establish the convergence of our joint learning process and provide error bounds between the discovered policy and the true optimal policy. We showcased our framework in grid-world environments, successfully identifying both acceptable safety constraints and RL policies while demonstrating the effectiveness of our theorems in practice.