# Transfer of Temporal Logic Formulas in Reinforcement Learning

arXiv: https://arxiv.org/abs/1909.04256 · HTML: https://mlsanity.com/paper/1909.04256/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 30 of 31)

- [LTL2Action: Generalizing LTL Instructions for Multi-Task RL](https://mlsanity.com/paper/2102.06858/) (2021)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees](https://mlsanity.com/paper/2305.01381/) (2023)
- [Active Task-Inference-Guided Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2001.09227/) (2020)
- [Policy Synthesis and Reinforcement Learning for Discounted LTL](https://mlsanity.com/paper/2305.17115/) (2023)
- [LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents](https://mlsanity.com/paper/2310.09454/) (2023)
- [Reward Machines for Deep RL in Noisy and Uncertain Environments](https://mlsanity.com/paper/2406.00120/) (2024)
- [Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines](https://mlsanity.com/paper/2211.10902/) (2022)
- [Specification-Guided Learning of Nash Equilibria with High Social Welfare](https://mlsanity.com/paper/2206.03348/) (2022)
- [Neural Network for Weighted Signal Temporal Logic](https://mlsanity.com/paper/2104.05435/) (2021)
- [Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines](https://mlsanity.com/paper/2111.09475/) (2021)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Synthesizing Efficiently Monitorable Formulas in Metric Temporal Logic](https://mlsanity.com/paper/2310.17410/) (2023)
- [Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications](https://mlsanity.com/paper/2311.17059/) (2023)
- [Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents](https://mlsanity.com/paper/2402.03678/) (2024)
- [Inductive Generalization in Reinforcement Learning from Specifications](https://mlsanity.com/paper/2406.03651/) (2024)
- [Reinforcement Learning with Temporal-Logic-Based Causal Diagrams](https://mlsanity.com/paper/2306.13732/) (2023)
- [HypRL: Reinforcement Learning of Control Policies for Hyperproperties](https://mlsanity.com/paper/2504.04675/) (2025)
- [Robust Pandemic Control Synthesis with Formal Specifications: A Case Study on COVID-19 Pandemic](https://mlsanity.com/paper/2103.14262/) (2021)
- [Reward-Machine-Guided, Self-Paced Reinforcement Learning](https://mlsanity.com/paper/2305.16505/) (2023)
- [Provably Correct Controller Synthesis of Switched Stochastic Systems with Metric Temporal Logic Specifications: A Case Study on Power Systems](https://mlsanity.com/paper/2103.14264/) (2021)
- [Reward Augmentation in Reinforcement Learning for Testing Distributed Systems](https://mlsanity.com/paper/2409.02137/) (2024)
- [Hyperproperty-Constrained Secure Reinforcement Learning](https://mlsanity.com/paper/2508.00106/) (2025)
- [Automating the Refinement of Reinforcement Learning Specifications](https://mlsanity.com/paper/2512.01047/) (2025)
- [About Time: Model-free Reinforcement Learning with Timed Reward Machines](https://mlsanity.com/paper/2512.17637/) (2025)
- [Reinforcement learning with timed constraints for robotics motion planning](https://mlsanity.com/paper/2601.00087/) (2025)
- [On Tackling Complex Tasks with Reward Machines and Signal Temporal Logics](https://mlsanity.com/paper/2604.14440/) (2026)
- [Reinforcement Learning for Reachability: Guaranteeing Asymptotic Optimality](https://mlsanity.com/paper/2605.24740/) (2026)
- [Decoupled Behavioral Cloning for Scalable Inductive Generalization in RL from Specifications](https://mlsanity.com/paper/2606.00838/) (2026)
- [HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2606.30966/) (2026)

## Abstract (arXiv)

Transferring high-level knowledge from a source task to a target task is an effective way to expedite reinforcement learning (RL). For example, propositional logic and first-order logic have been used as representations of such knowledge. We study the transfer of knowledge between tasks in which the timing of the events matters. We call such tasks temporal tasks. We concretize similarity between temporal tasks through a notion of logical transferability, and develop a transfer learning approach between different yet similar temporal tasks. We first propose an inference technique to extract metric interval temporal logic (MITL) formulas in sequential disjunctive normal form from labeled trajectories collected in RL of the two tasks. If logical transferability is identified through this inference, we construct a timed automaton for each sequential conjunctive subformula of the inferred MITL formulas from both tasks. We perform RL on the extended state which includes the locations and clock valuations of the timed automata for the source task. We then establish mappings between the corresponding components (clocks, locations, etc.) of the timed automata from the two tasks, and transfer the extended Q-functions based on the established mappings. Finally, we perform RL on the extended state for the target task, starting with the transferred extended Q-functions. Our results in two case studies show, depending on how similar the source task and the target task are, that the sampling efficiency for the target task can be improved by up to one order of magnitude by performing RL in the extended state space, and further improved by up to another order of magnitude using the transferred extended Q-functions.