mlsanityData to 14 Oct 2026

LTL-Constrained Policy Optimization with Cycle Experience Replay

arXiv 2404.11578 ↗First posted 17 Apr 2024cs.LG cs.AI cs.FLcited by 11

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

Linear Temporal Logic (LTL) offers a precise means for constraining the behavior of reinforcement learning agents. However, in many settings where both satisfaction and optimality conditions are present, LTL is insufficient to capture both. Instead, LTL-constrained policy optimization, where the goal is to optimize a scalar reward under LTL constraints, is needed. This constrained optimization problem proves difficult in deep Reinforcement Learning (DRL) settings, where learned policies often ignore the LTL constraint due to the sparse nature of LTL satisfaction. To alleviate the sparsity issue, we introduce Cycle Experience Replay (CyclER), a novel reward shaping technique that exploits the underlying structure of the LTL constraint to guide a policy towards satisfaction by encouraging partial behaviors compliant with the constraint. We provide a theoretical guarantee that optimizing CyclER will achieve policies that satisfy the LTL constraint with near-optimal probability. We evaluate CyclER in three continuous control domains. Our experimental results show that optimizing CyclER in tandem with the existing scalar reward outperforms existing reward-shaping methods at finding performant LTL-satisfying policies.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 53 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

31 of 53 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2019Batch Policy Learning under Constraintscited by 244
  5. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  6. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  7. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  8. 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 75
  9. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  10. 2021On the Expressivity of Markov Rewardcited by 71
  11. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 69
  12. 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 67
19 more
  1. 2021Compositional Reinforcement Learning from Logical Specificationscited by 67
  2. 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 54
  3. 2018Logically-Constrained Reinforcement Learningcited by 51
  4. 2014A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specificationscited by 49
  5. 2020Tractable Reinforcement Learning of Signal Temporal Logic Objectivescited by 22
  6. 2022From Spot 2.0 to Spot 2.10: What's New?cited by 22
  7. 2022Policy Optimization with Linear Temporal Logic Constraintscited by 19
  8. 2023Eventual Discounting Temporal Logic Counterfactual Experience Replaycited by 19
  9. 2021On the (In)Tractability of Reinforcement Learning for LTL Objectivescited by 18
  10. 2021In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specificationscited by 17
  11. 2020Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learningcited by 15
  12. 2020Optimal Probabilistic Motion Planning with Potential Infeasible LTL Constraintscited by 14
  13. 2023Policy Synthesis and Reinforcement Learning for Discounted LTLcited by 14
  14. 2023A PAC Learning Algorithm for LTL and Omega-regular Objectives in MDPscited by 13
  15. 2020Reinforcement Learning of Control Policy for Linear Temporal Logic Specifications Using Limit-Deterministic Generalized B"uchi Automatacited by 11
  16. 2020Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networkscited by 8
  17. 2022Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxationcited by 8
  18. 2020Event-Based Signal Temporal Logic Synthesis for Single and Multi-Robot Taskscited by 7
  19. 2022Safe Reinforcement Learning with Probabilistic Guarantees Satisfying Temporal Logic Specifications in Continuous Action Spacescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RLcited by 21
  2. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  3. 2025Automaton Constrained Q-Learningcited by 2
  4. 2026PlatoLTL: Scaling LTL-Guided Multi-Task RLcited by 2
  5. 2025Automata-Conditioned Cooperative Multi-Agent Reinforcement Learningcited by 1
  6. 2026Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 1
  7. 2026SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learningcited by 1
  8. 2025Near-Optimal Reinforcement Learning for Constrained Recurrence Objectivescited by 0
  9. 2025Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 0
  10. 2026Live LTL Progress Tracking: Towards Task-Based Explorationcited by 0
  11. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerated Learning with Linear Temporal Logic using Differentiable Simulation
  2. 2024Adaptive Reward Design for Reinforcement Learning
  3. 2024Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching
  4. 2024Reinforcement Learning with LTL and ω-Regular Objectives via Optimality-Preserving Translation to Average Rewards
  5. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logic
  6. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control
  7. 2024Regret-Free Reinforcement Learning for LTL Specifications
  8. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients