# STL-Based Synthesis of Feedback Controllers Using Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.01022 · HTML: https://mlsanity.com/paper/2212.01022/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 10 of 10)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [On Robustness Metrics for Learning STL Tasks](https://mlsanity.com/paper/2003.06041/) (2020)
- [A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks](https://mlsanity.com/paper/1709.09611/) (2017)
- [Arithmetic-Geometric Mean Robustness for Control from Signal Temporal Logic Specifications](https://mlsanity.com/paper/1903.05186/) (2019)
- [RTAMT: Online Robustness Monitors from STL](https://mlsanity.com/paper/2005.11827/) (2020)
- [Smoothed Action Value Functions for Learning Gaussian Policies](https://mlsanity.com/paper/1803.02348/) (2018)

## Built on by (most cited first, 4 of 4)

- [Multi-agent transformer-accelerated RL for satisfaction of STL specifications](https://mlsanity.com/paper/2403.15916/) (2024)
- [Conformal Signal Temporal Logic for Robust Reinforcement Learning Control: A Case Study](https://mlsanity.com/paper/2602.14322/) (2026)
- [Hyperproperty-Constrained Secure Reinforcement Learning](https://mlsanity.com/paper/2508.00106/) (2025)
- [Control Synthesis of Cyber-Physical Systems for Real-Time Specifications through Causation-Guided Reinforcement Learning](https://mlsanity.com/paper/2510.07715/) (2025)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) has the potential to be used for synthesizing feedback controllers (agents) for various complex systems with unknown dynamics. These systems are expected to satisfy diverse safety and liveness properties best captured using temporal logic. In RL, the reward function plays a crucial role in specifying the desired behaviour of these agents. However, the problem of designing the reward function for an RL agent to satisfy complex temporal logic specifications has received limited attention in the literature. To address this, we provide a systematic way of generating rewards in real-time by using the quantitative semantics of Signal Temporal Logic (STL), a widely used temporal logic to specify the behaviour of cyber-physical systems. We propose a new quantitative semantics for STL having several desirable properties, making it suitable for reward generation. We evaluate our STL-based reinforcement learning mechanism on several complex continuous control benchmarks and compare our STL semantics with those available in the literature in terms of their efficacy in synthesizing the controller agent. Experimental results establish our new semantics to be the most suitable for synthesizing feedback controllers for complex continuous dynamical systems through reinforcement learning.