mlsanityData to 7 Oct 2026

Logically-Constrained Reinforcement Learning

arXiv 1801.08099 ↗First posted 24 Jan 2018cs.LG cs.LOcited by 50

Not on the map. Why?

Abstract (arXiv)

We present the first model-free Reinforcement Learning (RL) algorithm to synthesise policies for an unknown Markov Decision Process (MDP), such that a linear time property is satisfied. The given temporal property is converted into a Limit Deterministic Buchi Automaton (LDBA) and a robust reward function is defined over the state-action pairs of the MDP according to the resulting LDBA. With this reward function, the policy synthesis procedure is "constrained" by the given specification. These constraints guide the MDP exploration so as to minimize the solution time by only considering the portion of the MDP that is relevant to satisfaction of the LTL property. This improves performance and scalability of the proposed method by avoiding an exhaustive update over the whole state space while the efficiency of standard methods such as dynamic programming is hindered by excessive memory requirements, caused by the need to store a full-model in memory. Additionally, we show that the RL procedure sets up a local value iteration method to efficiently calculate the maximum probability of satisfying the given property, at any given state of the MDP. We prove that our algorithm is guaranteed to find a policy whose traces probabilistically satisfy the LTL property if such a policy exists, and additionally we show that our method produces reasonable control policies even when the LTL property cannot be satisfied. The performance of the algorithm is evaluated via a set of numerical examples. We observe an improvement of one order of magnitude in the number of iterations required for the synthesis compared to existing approaches.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 10; 2024: 7; 2025: 6; 2026: 9.

50 papers.

Sort:

By area:

  1. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  2. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  3. 2021Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logiccited by 38
  4. 2021The Logical Options Frameworkcited by 24
  5. 2024DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RLcited by 20
  6. 2022Policy Optimization with Linear Temporal Logic Constraintscited by 19
  7. 2023Eventual Discounting Temporal Logic Counterfactual Experience Replaycited by 18
  8. 2023ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learningcited by 15
  9. 2023Policy Synthesis and Reinforcement Learning for Discounted LTLcited by 14
  10. 2022Joint Learning of Reward Machines and Policies in Environments with Partially Known Semanticscited by 12
  11. 2022LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learningcited by 11
  12. 2024Reward Machines for Deep RL in Noisy and Uncertain Environmentscited by 11
38 more
  1. 2021Reinforcement Learning with External Knowledge by using Logical Neural Networkscited by 10
  2. 2022Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machinescited by 10
  3. 2024LTL-Constrained Policy Optimization with Cycle Experience Replaycited by 10
  4. 2022Accelerated Reinforcement Learning for Temporal Logic Control Objectivescited by 9
  5. 2022Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxationcited by 8
  6. 2022Specification-Guided Learning of Nash Equilibria with High Social Welfarecited by 8
  7. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  8. 2022Learning Task Automata for Reinforcement Learning using Hidden Markov Modelscited by 7
  9. 2024Directed Exploration in Reinforcement Learning from Linear Temporal Logiccited by 7
  10. 2022Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectivescited by 6
  11. 2023Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Surveycited by 5
  12. 2022Possibility Before Utility: Learning And Using Hierarchical Affordancescited by 4
  13. 2022Learning Globally Smooth Functions on Manifoldscited by 4
  14. 2023Learning Symbolic Representations for Reinforcement Learning of Non-Markovian Behaviorcited by 4
  15. 2024Inductive Generalization in Reinforcement Learning from Specificationscited by 3
  16. 2021Learning to Act Safely with Limited Exposure and Almost Sure Certaintycited by 2
  17. 2021On Assessing The Safety of Reinforcement Learning algorithms Using Formal Methodscited by 2
  18. 2023System III: Learning with Domain Knowledge for Safety Constraintscited by 2
  19. 2025HypRL: Reinforcement Learning of Control Policies for Hyperpropertiescited by 2
  20. 2023Learning Density-Based Correlated Equilibria for Markov Gamescited by 1
  21. 2023Joint Learning of Policy with Unknown Temporal Constraints for Safe Reinforcement Learningcited by 1
  22. 2023Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesiscited by 1
  23. 2026Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 1
  24. 2026hint2: Hierarchical World Models for Inference-Time Temporal Logic Guidancecited by 1
  25. 2023Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)cited by 0
  26. 2024Scaling Learning based Policy Optimization for Temporal Logic Tasks by Controller Network Dropoutcited by 0
  27. 2024Progressive Safeguards for Safe and Model-Agnostic Reinforcement Learningcited by 0
  28. 2025Situational-Constrained Sequential Resources Allocation via Reinforcement Learningcited by 0
  29. 2025Constraints-Guided Diffusion Reasoner for Neuro-Symbolic Learningcited by 0
  30. 2025RLAF: Reinforcement Learning from Automaton Feedbackcited by 0
  31. 2025Zero-Shot Instruction Following in RL via Structured LTL Representationscited by 0
  32. 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
  33. 2026Reinforcement Learning for Reachability: Guaranteeing Asymptotic Optimalitycited by 0
  34. 2026Decoupled Behavioral Cloning for Scalable Inductive Generalization in RL from Specificationscited by 0
  35. 2026REBA: A Revealed Belief Automaton Framework for Online Planning in Continuous POMDPscited by 0
  36. 2026HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observationcited by 0
  37. 2026Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specificationscited by 0
  38. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0