# Reinforcement Learning with Temporal-Logic-Based Causal Diagrams

arXiv: https://arxiv.org/abs/2306.13732 · HTML: https://mlsanity.com/paper/2306.13732/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 16 of 16)

- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [Causal Bandits: Learning Good Interventions via Causal Inference](https://mlsanity.com/paper/1606.03203/) (2016)
- [Counterfactual Data Augmentation using Locally Factored Dynamics](https://mlsanity.com/paper/2007.02863/) (2020)
- [Q-Learning for Robust Satisfaction of Signal Temporal Logic Specifications](https://mlsanity.com/paper/1609.07409/) (2016)
- [Off-Policy Evaluation in Partially Observable Environments](https://mlsanity.com/paper/1909.03739/) (2019)
- [Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation](https://mlsanity.com/paper/2012.09092/) (2020)
- [Joint Inference of Reward Machines and Policies for Reinforcement Learning](https://mlsanity.com/paper/1909.05912/) (2019)
- [Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints](https://mlsanity.com/paper/1404.7073/) (2014)
- [AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning](https://mlsanity.com/paper/2107.02729/) (2021)
- [Transfer of Temporal Logic Formulas in Reinforcement Learning](https://mlsanity.com/paper/1909.04256/) (2019)
- [Learning Causal State Representations of Partially Observable Environments](https://mlsanity.com/paper/1906.10437/) (2019)
- [Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples](https://mlsanity.com/paper/2006.15714/) (2020)
- [Symbolic LTLf Synthesis](https://mlsanity.com/paper/1705.08426/) (2017)
- [Active Task-Inference-Guided Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2001.09227/) (2020)
- [Learning Task Automata for Reinforcement Learning using Hidden Markov Models](https://mlsanity.com/paper/2208.11838/) (2022)

## Built on by (most cited first, 2 of 2)

- [Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environment](https://mlsanity.com/paper/2510.15456/) (2025)
- [Decentralizing Multi-Agent Reinforcement Learning with Temporal Causal Information](https://mlsanity.com/paper/2506.07829/) (2025)

## Abstract (arXiv)

We study a class of reinforcement learning (RL) tasks where the objective of the agent is to accomplish temporally extended goals. In this setting, a common approach is to represent the tasks as deterministic finite automata (DFA) and integrate them into the state-space for RL algorithms. However, while these machines model the reward function, they often overlook the causal knowledge about the environment. To address this limitation, we propose the Temporal-Logic-based Causal Diagram (TL-CD) in RL, which captures the temporal causal relationships between different properties of the environment. We exploit the TL-CD to devise an RL algorithm in which an agent requires significantly less exploration of the environment. To this end, based on a TL-CD and a task DFA, we identify configurations where the agent can determine the expected rewards early during an exploration. Through a series of case studies, we demonstrate the benefits of using TL-CDs, particularly the faster convergence of the algorithm to an optimal policy due to reduced exploration of the environment.