# Cautious Reinforcement Learning with Logical Constraints

arXiv: https://arxiv.org/abs/2002.12156 · HTML: https://mlsanity.com/paper/2002.12156/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 30 of 45)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Safe Multi-Agent Reinforcement Learning via Shielding](https://mlsanity.com/paper/2101.11196/) (2021)
- [Safe Reinforcement Learning via Shielding under Partial Observability](https://mlsanity.com/paper/2204.00755/) (2022)
- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic](https://mlsanity.com/paper/2102.12855/) (2021)
- [Safe Reinforcement Learning for Legged Locomotion](https://mlsanity.com/paper/2203.02638/) (2022)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [Learning Control Policies for Stochastic Systems with Reach-avoid Guarantees](https://mlsanity.com/paper/2210.05308/) (2022)
- [Online Shielding for Reinforcement Learning](https://mlsanity.com/paper/2212.01861/) (2022)
- [ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning](https://mlsanity.com/paper/2309.13701/) (2023)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Shielding Atari Games with Bounded Prescience](https://mlsanity.com/paper/2101.08153/) (2021)
- [Correct-by-Construction Runtime Enforcement in AI -- A Survey](https://mlsanity.com/paper/2208.14426/) (2022)
- [Dynamic Shielding for Reinforcement Learning in Black-Box Environments](https://mlsanity.com/paper/2207.13446/) (2022)
- [LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning](https://mlsanity.com/paper/2209.10341/) (2022)
- [Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.06281/) (2023)
- [Reinforcement Learning with External Knowledge by using Logical Neural Networks](https://mlsanity.com/paper/2103.02363/) (2021)
- [Shielded Reinforcement Learning for Hybrid Systems](https://mlsanity.com/paper/2308.14424/) (2023)
- [Verified Probabilistic Policies for Deep Reinforcement Learning](https://mlsanity.com/paper/2201.03698/) (2022)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environments](https://mlsanity.com/paper/2303.14265/) (2023)
- [Optimal Transport-Assisted Risk-Sensitive Q-Learning](https://mlsanity.com/paper/2406.11774/) (2024)
- [Reinforcement Learning Guided by Provable Normative Compliance](https://mlsanity.com/paper/2203.16275/) (2022)
- [Logic-based Reward Shaping for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.08881/) (2022)
- [Tractable Offline Learning of Regular Decision Processes](https://mlsanity.com/paper/2409.02747/) (2024)
- [System III: Learning with Domain Knowledge for Safety Constraints](https://mlsanity.com/paper/2304.11593/) (2023)
- [Reinforcement Learning Under Probabilistic Spatio-Temporal Constraints with Time Windows](https://mlsanity.com/paper/2307.15910/) (2023)
- [Composing Reinforcement Learning Policies, with Formal Guarantees](https://mlsanity.com/paper/2402.13785/) (2024)
- [Automaton Constrained Q-Learning](https://mlsanity.com/paper/2510.05061/) (2025)

## Abstract (arXiv)

This paper presents the concept of an adaptive safe padding that forces Reinforcement Learning (RL) to synthesise optimal control policies while ensuring safety during the learning process. Policies are synthesised to satisfy a goal, expressed as a temporal logic formula, with maximal probability. Enforcing the RL agent to stay safe during learning might limit the exploration, however we show that the proposed architecture is able to automatically handle the trade-off between efficient progress in exploration (towards goal satisfaction) and ensuring safety. Theoretical guarantees are available on the optimality of the synthesised policies and on the convergence of the learning algorithm. Experimental results are provided to showcase the performance of the proposed method.