# Reinforcement Learning Under Probabilistic Spatio-Temporal Constraints with Time Windows

arXiv: https://arxiv.org/abs/2307.15910 · HTML: https://mlsanity.com/paper/2307.15910/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 29 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Cautious Reinforcement Learning with Logical Constraints](https://mlsanity.com/paper/2002.12156/) (2020)
- [Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPs](https://mlsanity.com/paper/2008.00311/) (2020)
- [Probabilistic Model Checking of Robots Deployed in Extreme Environments](https://mlsanity.com/paper/1812.04128/) (2018)
- [Probabilistically Guaranteed Satisfaction of Temporal Logic Constraints During Reinforcement Learning](https://mlsanity.com/paper/2102.10063/) (2021)
- [Time Window Temporal Logic](https://mlsanity.com/paper/1602.04294/) (2016)
- [Monitoring Bounded LTL Properties Using Interval Analysis](https://mlsanity.com/paper/1506.01762/) (2015)

## Built on by (most cited first, 2 of 2)

- [Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching](https://mlsanity.com/paper/2410.08022/) (2024)
- [Shielded Reinforcement Learning Under Dynamic Temporal Logic Constraints](https://mlsanity.com/paper/2603.17152/) (2026)

## Abstract (arXiv)

We propose an automata-theoretic approach for reinforcement learning (RL) under complex spatio-temporal constraints with time windows. The problem is formulated using a Markov decision process under a bounded temporal logic constraint. Different from existing RL methods that can eventually learn optimal policies satisfying such constraints, our proposed approach enforces a desired probability of constraint satisfaction throughout learning. This is achieved by translating the bounded temporal logic constraint into a total automaton and avoiding "unsafe" actions based on the available prior information regarding the transition probabilities, i.e., a pair of upper and lower bounds for each transition probability. We provide theoretical guarantees on the resulting probability of constraint satisfaction. We also provide numerical results in a scenario where a robot explores the environment to discover high-reward regions while fulfilling some periodic pick-up and delivery tasks that are encoded as temporal logic constraints.