mlsanityData to 7 Oct 2026

Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies

arXiv 2606.08312 ↗First posted 6 Jun 2026cs.AI cs.FLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this work we study offline reinforcement learning (RL) under temporally extended task constraints expressed in Linear Temporal Logic over finite traces (LTLf). Recently, transformer-based approaches such as Trajectory Transformers and Decision Transformers have been adopted to address RL as a sequence modeling problem. However, these methods optimize purely for reward and do not account for high-level temporal requirements. Here, we introduce a neurosymbolic framework that injects LTLf background knowledge into such transformer-based RL policies. Our approach compiles LTLf formulas into deterministic finite automata (DFAs) and integrates them into the learning process through a differentiable representation and a logic-based loss function. In particular, we derive differentiable satisfaction signals from DFA progression and use them as a regularization term during training. The resulting method is architecture-agnostic across different models. We evaluate the proposed framework on navigation environments with specification suites covering combinations of safety and reachability temporal properties. Experimental results show that incorporating background knowledge not only improves constraint satisfaction, but also maintains competitive return compared to vanilla baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 52 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

37 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  2. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2017Safe Reinforcement Learning via Shieldingcited by 427
  6. 2018Reward Constrained Policy Optimizationcited by 397
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2022Online Decision Transformercited by 194
  9. 2019Goal-conditioned Imitation Learningcited by 159
  10. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  11. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  12. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
21 more
  1. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  2. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  3. 2019Reward-Conditioned Policiescited by 101
  4. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  5. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  6. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  7. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  8. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  9. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 41
  10. 2024Adaptable Logical Control for Large Language Modelscited by 30
  11. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  12. 2023A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraintscited by 21
  13. 2020Elaborating on Learned Demonstrations with Temporal Logic Specificationscited by 14
  14. 2025Probabilistic Shielding for Safe Reinforcement Learningcited by 12
  15. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 12
  16. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  17. 2024NeSyA: Neurosymbolic Automatacited by 9
  18. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RLcited by 8
  19. 2024DeepDFA: Automata Learning through Neural Probabilistic Relaxationscited by 8
  20. 2018Safe Reinforcement Learning via Probabilistic Shieldscited by 7
  21. 2025Neuro-Symbolic Predictive Process Monitoringcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  2. 2025RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
  3. 2025Expressive Value Learning for Scalable Offline Reinforcement Learning
  4. 2025RLAF: Reinforcement Learning from Automaton Feedback
  5. 2026Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
  6. 2025Zero-Shot Instruction Following in RL via Structured LTL Representations
  7. 2026Time to Reason: Scalable Neurosymbolic Learning for LTLf via Fuzzy Semantics
  8. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning