mlsanityData to 14 Oct 2026

Accelerated Learning with Linear Temporal Logic using Differentiable Simulation

arXiv 2506.01167 ↗First posted 1 Jun 2025cs.LG cs.ROcited by 1

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

Ensuring that reinforcement learning (RL) controllers satisfy safety and reliability constraints in real-world settings remains challenging: state-avoidance and constrained Markov decision processes often fail to capture trajectory-level requirements or induce overly conservative behavior. Formal specification languages such as linear temporal logic (LTL) offer correct-by-construction objectives, yet their rewards are typically sparse, and heuristic shaping can undermine correctness. We introduce, to our knowledge, the first end-to-end framework that integrates LTL with differentiable simulators, enabling efficient gradient-based learning directly from formal specifications. Our method relaxes discrete automaton transitions via soft labeling of states, yielding differentiable rewards and state representations that mitigate the sparsity issue intrinsic to LTL while preserving objective soundness. We provide theoretical guarantees connecting B"uchi acceptance to both discrete and differentiable LTL returns and derive a tunable bound on their discrepancy in deterministic and stochastic settings. Empirically, across complex, nonlinear, contact-rich continuous-control tasks, our approach substantially accelerates training and achieves up to twice the returns of discrete baselines. We further demonstrate compatibility with reward machines, thereby covering co-safe LTL and LTLf without modification. By rendering automaton-based rewards differentiable, our work bridges formal methods and deep RL, enabling safe, specification-driven learning in continuous domains.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 73 of its 114 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

73 of 114 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learning31 Oct
  2. 2024Reinforcement Learning with LTL and ω-Regular Objectives via Optimality-Preserving Translation to Average Rewards16 Oct
  3. 2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL6 Oct

Other references · 70

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  6. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  7. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  8. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  9. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  10. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  11. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  12. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
38 more
  1. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  2. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  3. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  4. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  5. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  6. 2022Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methodscited by 151
  7. 2017Socially Aware Motion Planning with Deep Reinforcement Learningcited by 136
  8. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  9. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  10. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 110
  11. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  12. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 103
  13. 2018ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Roboticscited by 98
  14. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  15. 2021Gradients are Not All You Needcited by 86
  16. 2019Safe Reinforcement Learning Using Robust MPCcited by 85
  17. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  18. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  19. 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 75
  20. 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 75
  21. 2021Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificatescited by 74
  22. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 72
  23. 2020Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functionscited by 68
  24. 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 67
  25. 2021DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cuttingcited by 64
  26. 2021DiffPD: Differentiable Projective Dynamicscited by 62
  27. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  28. 2022Reachability Constrained Reinforcement Learningcited by 59
  29. 2018Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networkscited by 56
  30. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  31. 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 54
  32. 2020Backpropagation through Signal Temporal Logic Specifications: Infusing Logical Structure into Gradient-Based Methodscited by 54
  33. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 54
  34. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 52
  35. 2021Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learningcited by 49
  36. 2019Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamicscited by 47
  37. 2014Verification of Markov Decision Processes using Learning Algorithmscited by 39
  38. 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 39

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reward Machines for Signal Temporal Logiccited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logic
  2. 2026Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies
  3. 2024Regret-Free Reinforcement Learning for LTL Specifications
  4. 2024Sample-Efficient Reinforcement Learning with Temporal Logic Objectives: Leveraging the Task Specification to Guide Exploration
  5. 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systems
  6. 2025Automaton Constrained Q-Learning
  7. 2026Do It for HER: First-Order Temporal Logic Reward Specification in Reinforcement Learning (Extended Version)
  8. 2024Comp-LTL: Temporal Logic Planning via Zero-Shot Policy Composition