# TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks

arXiv: https://arxiv.org/abs/2510.00225 · HTML: https://mlsanity.com/paper/2510.00225/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [A Composable Specification Language for Reinforcement Learning Tasks](https://mlsanity.com/paper/2008.09293/) (2020)
- [Q-Learning for Robust Satisfaction of Signal Temporal Logic Specifications](https://mlsanity.com/paper/1609.07409/) (2016)
- [Backpropagation through Signal Temporal Logic Specifications: Infusing Logical Structure into Gradient-Based Methods](https://mlsanity.com/paper/2008.00097/) (2020)
- [A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specifications](https://mlsanity.com/paper/1409.5486/) (2014)
- [Tractable Reinforcement Learning of Signal Temporal Logic Objectives](https://mlsanity.com/paper/2001.09467/) (2020)
- [Multi-agent Motion Planning from Signal Temporal Logic Specifications](https://mlsanity.com/paper/2201.05247/) (2022)
- [Recurrent Neural Network Controllers for Signal Temporal Logic Specifications Subject to Safety Constraints](https://mlsanity.com/paper/2009.11468/) (2020)
- [Model-based Reinforcement Learning from Signal Temporal Logic Specifications](https://mlsanity.com/paper/2011.04950/) (2020)
- [Learning from Demonstrations using Signal Temporal Logic](https://mlsanity.com/paper/2102.07730/) (2021)
- [Mixed-Integer Programming for Signal Temporal Logic with Fewer Binary Variables](https://mlsanity.com/paper/2204.06367/) (2022)
- [Signal Temporal Logic Neural Predictive Control](https://mlsanity.com/paper/2309.05131/) (2023)
- [Semi-Supervised Trajectory-Feedback Controller Synthesis for Signal Temporal Logic Specifications](https://mlsanity.com/paper/2202.01997/) (2022)
- [TeLoGraF: Temporal Logic Planning via Graph-encoded Flow Matching](https://mlsanity.com/paper/2505.00562/) (2025)
- [Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation](https://mlsanity.com/paper/2201.08504/) (2022)
- [Diverse Controllable Diffusion Policy with Signal Temporal Logic](https://mlsanity.com/paper/2503.02924/) (2025)
- [Learning Robust and Correct Controllers from Signal Temporal Logic Specifications Using BarrierNet](https://mlsanity.com/paper/2304.06160/) (2023)
- [Safe Planning through Incremental Decomposition of Signal Temporal Logic Specifications](https://mlsanity.com/paper/2403.10554/) (2024)
- [Zero-Shot Trajectory Planning for Signal Temporal Logic Tasks](https://mlsanity.com/paper/2501.13457/) (2025)

## Built on by (most cited first, 6 of 6)

- [SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning](https://mlsanity.com/paper/2604.24729/) (2026)
- [ReasonSTL: Bridging Natural Language and Signal Temporal Logic via Tool-Augmented Process-Rewarded Learning](https://mlsanity.com/paper/2605.06483/) (2026)
- [Feasibility-aware Learning of Robust Temporal Logic Controllers using BarrierNet](https://mlsanity.com/paper/2512.06973/) (2025)
- [Zero-Shot Signal Temporal Logic Planning with Disjunctive Branch Selection in Dynamic Semantic Maps](https://mlsanity.com/paper/2605.01222/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient](https://mlsanity.com/paper/2609.31606/) (2026)

## Abstract (arXiv)

Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at https://github.com/mengyuest/TGPO