# Q-Learning for Robust Satisfaction of Signal Temporal Logic Specifications

arXiv: https://arxiv.org/abs/1609.07409 · HTML: https://mlsanity.com/paper/1609.07409/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2016.

## Built on by (most cited first, 30 of 67)

- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic](https://mlsanity.com/paper/2102.12855/) (2021)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [DeepSTL -- From English Requirements to Signal Temporal Logic](https://mlsanity.com/paper/2109.10294/) (2021)
- [Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2411.00205/) (2024)
- [Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees](https://mlsanity.com/paper/2305.01381/) (2023)
- [Learning from Demonstrations using Signal Temporal Logic](https://mlsanity.com/paper/2102.07730/) (2021)
- [Policy Synthesis and Reinforcement Learning for Discounted LTL](https://mlsanity.com/paper/2305.17115/) (2023)
- [Signal Temporal Logic Neural Predictive Control](https://mlsanity.com/paper/2309.05131/) (2023)
- [Joint Learning of Reward Machines and Policies in Environments with Partially Known Semantics](https://mlsanity.com/paper/2204.11833/) (2022)
- [LTL-Constrained Policy Optimization with Cycle Experience Replay](https://mlsanity.com/paper/2404.11578/) (2024)
- [Reward Machines for Deep RL in Noisy and Uncertain Environments](https://mlsanity.com/paper/2406.00120/) (2024)
- [Model-Free Reinforcement Learning for Optimal Control of MarkovDecision Processes Under Signal Temporal Logic Specifications](https://mlsanity.com/paper/2109.13377/) (2021)
- [Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines](https://mlsanity.com/paper/2211.10902/) (2022)
- [Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation](https://mlsanity.com/paper/2201.08504/) (2022)
- [Specification-Guided Learning of Nash Equilibria with High Social Welfare](https://mlsanity.com/paper/2206.03348/) (2022)
- [Funnel-based Reward Shaping for Signal Temporal Logic Tasks in Reinforcement Learning](https://mlsanity.com/paper/2212.03181/) (2022)
- [Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectives](https://mlsanity.com/paper/2202.02404/) (2022)
- [Learning Robust and Correct Controllers from Signal Temporal Logic Specifications Using BarrierNet](https://mlsanity.com/paper/2304.06160/) (2023)
- [Zero-Shot Trajectory Planning for Signal Temporal Logic Tasks](https://mlsanity.com/paper/2501.13457/) (2025)
- [TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks](https://mlsanity.com/paper/2510.00225/) (2025)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data](https://mlsanity.com/paper/2507.10741/) (2025)
- [Logically Constrained Robotics Transformers for Enhanced Perception-Action Planning](https://mlsanity.com/paper/2408.05336/) (2024)
- [Safe Model-based Control from Signal Temporal Logic Specifications Using Recurrent Neural Networks](https://mlsanity.com/paper/2103.15938/) (2021)
- [Reinforcement Learning with Formal Performance Metrics for Quadcopter Attitude Control under Non-nominal Contexts](https://mlsanity.com/paper/2107.12942/) (2021)
- [Learning Performance Graphs from Demonstrations via Task-Based Evaluations](https://mlsanity.com/paper/2204.05909/) (2022)
- [Interpretable Imitation Learning via Generative Adversarial STL Inference and Control](https://mlsanity.com/paper/2402.10310/) (2024)

## Abstract (arXiv)

This paper addresses the problem of learning optimal policies for satisfying signal temporal logic (STL) specifications by agents with unknown stochastic dynamics. The system is modeled as a Markov decision process, in which the states represent partitions of a continuous space and the transition probabilities are unknown. We formulate two synthesis problems where the desired STL specification is enforced by maximizing the probability of satisfaction, and the expected robustness degree, that is, a measure quantifying the quality of satisfaction. We discuss that Q-learning is not directly applicable to these problems because, based on the quantitative semantics of STL, the probability of satisfaction and expected robustness degree are not in the standard objective form of Q-learning. To resolve this issue, we propose an approximation of STL synthesis problems that can be solved via Q-learning, and we derive some performance bounds for the policies obtained by the approximate approach. The performance of the proposed method is demonstrated via simulations.