Reinforcement Learning with Temporal-Logic-Based Causal Diagrams
Not on the map. Why?
Abstract (arXiv)
We study a class of reinforcement learning (RL) tasks where the objective of the agent is to accomplish temporally extended goals. In this setting, a common approach is to represent the tasks as deterministic finite automata (DFA) and integrate them into the state-space for RL algorithms. However, while these machines model the reward function, they often overlook the causal knowledge about the environment. To address this limitation, we propose the Temporal-Logic-based Causal Diagram (TL-CD) in RL, which captures the temporal causal relationships between different properties of the environment. We exploit the TL-CD to devise an RL algorithm in which an agent requires significantly less exploration of the environment. To this end, based on a TL-CD and a task DFA, we identify configurations where the agent can determine the expected rewards early during an exploration. Through a series of case studies, we demonstrate the benefits of using TL-CDs, particularly the faster convergence of the algorithm to an optimal policy due to reduced exploration of the environment.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 16 of its 31 references that are in the map. First posted on arXiv 23 Jun 2023.
Builds on
16 of 31 references.
References · 16
- 2017Safe Reinforcement Learning via Shieldingcited by 449
- 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
- 2016Causal Bandits: Learning Good Interventions via Causal Inferencecited by 101
- 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
- 2016Q-Learning for Robust Satisfaction of Signal Temporal Logic Specificationscited by 67
- 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
- 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 59
- 2019Joint Inference of Reward Machines and Policies for Reinforcement Learningcited by 56
- 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 54
- 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 48
- 2019Transfer of Temporal Logic Formulas in Reinforcement Learningcited by 31
- 2019Learning Causal State Representations of Partially Observable Environmentscited by 30
4 more
- 2020Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamplescited by 23
- 2017Symbolic LTLf Synthesiscited by 21
- 2020Active Task-Inference-Guided Deep Inverse Reinforcement Learningcited by 17
- 2022Learning Task Automata for Reinforcement Learning using Hidden Markov Modelscited by 7
Built on by
Citing papers per year: 2025: 2.
2 papers.