# Scaling Learning based Policy Optimization for Temporal Logic Tasks by Controller Network Dropout

arXiv: https://arxiv.org/abs/2403.15826 · HTML: https://mlsanity.com/paper/2403.15826/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 80 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Builds on (most cited first, 30 of 35)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems](https://mlsanity.com/paper/2004.05519/) (2020)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning](https://mlsanity.com/paper/1909.07299/) (2019)
- [Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints](https://mlsanity.com/paper/1404.7073/) (2014)
- [Backpropagation through Signal Temporal Logic Specifications: Infusing Logical Structure into Gradient-Based Methods](https://mlsanity.com/paper/2008.00097/) (2020)
- [Logically-Constrained Reinforcement Learning](https://mlsanity.com/paper/1801.08099/) (2018)
- [A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specifications](https://mlsanity.com/paper/1409.5486/) (2014)
- [Robustness of Control Barrier Functions for Safety Critical Control](https://mlsanity.com/paper/1612.01554/) (2016)
- [Tractable Reinforcement Learning of Signal Temporal Logic Objectives](https://mlsanity.com/paper/2001.09467/) (2020)
- [Robust Temporal Logic Model Predictive Control](https://mlsanity.com/paper/1511.00347/) (2015)
- [A Smooth Robustness Measure of Signal Temporal Logic for Symbolic Control](https://mlsanity.com/paper/2006.05239/) (2020)
- [Recurrent Neural Network Controllers for Signal Temporal Logic Specifications Subject to Safety Constraints](https://mlsanity.com/paper/2009.11468/) (2020)
- [Model-based Reinforcement Learning from Signal Temporal Logic Specifications](https://mlsanity.com/paper/2011.04950/) (2020)
- [Mixed-Integer Programming for Signal Temporal Logic with Fewer Binary Variables](https://mlsanity.com/paper/2204.06367/) (2022)
- [Learning Robust Hybrid Control Barrier Functions for Uncertain Systems](https://mlsanity.com/paper/2101.06492/) (2021)
- [Probabilistic Motion Planning under Temporal Tasks and Soft Constraints](https://mlsanity.com/paper/1706.05209/) (2017)
- [Time Robustness in MTL and Expressivity in Hybrid System Falsification (Extended Version)](https://mlsanity.com/paper/1505.06307/) (2015)
- [Gray-box Adversarial Testing for Control Systems with Machine Learning Component](https://mlsanity.com/paper/1812.11958/) (2018)
- [STL2vec: Signal Temporal Logic Embeddings for Control Synthesis With Recurrent Neural Networks](https://mlsanity.com/paper/2109.04636/) (2021)
- [Hierarchical Potential-based Reward Shaping from Task Specifications](https://mlsanity.com/paper/2110.02792/) (2021)
- [Synthesis of Discounted-Reward Optimal Policies for Markov Decision Processes Under Linear Temporal Logic Specifications](https://mlsanity.com/paper/2011.00632/) (2020)
- [Learning Performance Graphs from Demonstrations via Task-Based Evaluations](https://mlsanity.com/paper/2204.05909/) (2022)
- [A Neurosymbolic Approach to the Verification of Temporal Logic Properties of Learning enabled Control Systems](https://mlsanity.com/paper/2303.05394/) (2023)

## Abstract (arXiv)

This paper introduces a model-based approach for training feedback controllers for an autonomous agent operating in a highly nonlinear (albeit deterministic) environment. We desire the trained policy to ensure that the agent satisfies specific task objectives and safety constraints, both expressed in Discrete-Time Signal Temporal Logic (DT-STL). One advantage for reformulation of a task via formal frameworks, like DT-STL, is that it permits quantitative satisfaction semantics. In other words, given a trajectory and a DT-STL formula, we can compute the em robustness, which can be interpreted as an approximate signed distance between the trajectory and the set of trajectories satisfying the formula. We utilize feedback control, and we assume a feed forward neural network for learning the feedback controller. We show how this learning problem is similar to training recurrent neural networks (RNNs), where the number of recurrent units is proportional to the temporal horizon of the agent's task objectives. This poses a challenge: RNNs are susceptible to vanishing and exploding gradients, and na"ive gradient descent-based strategies to solve long-horizon task objectives thus suffer from the same problems. To tackle this challenge, we introduce a novel gradient approximation algorithm based on the idea of dropout or gradient sampling. One of the main contributions is the notion of em controller network dropout, where we approximate the NN controller in several time-steps in the task horizon by the control input obtained using the controller in a previous training step. We show that our control synthesis methodology, can be quite helpful for stochastic gradient descent to converge with less numerical issues, enabling scalable backpropagation over long time horizons and trajectories over high dimensional state spaces.