Harnessing Causality in Reinforcement Learning With Bagged Decision Times
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We consider reinforcement learning (RL) for a class of problems with bagged decision times. A bag contains a finite sequence of consecutive decision times. The transition dynamics are non-Markovian and non-stationary within a bag. All actions within a bag jointly impact a single reward, observed at the end of the bag. For example, in mobile health, multiple activity suggestions in a day collectively affect a user's daily commitment to being active. Our goal is to develop an online RL algorithm to maximize the discounted sum of the bag-specific rewards. To handle non-Markovian transitions within a bag, we utilize an expert-provided causal directed acyclic graph (DAG). Based on the DAG, we construct states as a dynamical Bayesian sufficient statistic of the observed history, which results in Markov state transitions within and across bags. We then formulate this problem as a periodic Markov decision process (MDP) that allows non-stationarity within a period. An online RL algorithm based on Bellman equations for stationary MDPs is generalized to handle periodic MDPs. We show that our constructed state achieves the maximal optimal value function among all state constructions for a periodic MDP. Finally, we evaluate the proposed method on testbed variants built from real data in a mobile health clinical trial.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 18 Oct 2024.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 3; 2026: 2.
5 papers.
- 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
- 2025Reinforcement Learning with Segment Feedbackcited by 4
- 2025Active Measuring in Reinforcement Learning With Delayed Negative Effectscited by 1
- 2025Generalized Linear Markov Decision Processcited by 0
- 2026From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RLcited by 0
- 2026From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RL8 May 2026
- 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions21 Jan 2026
- 2025Active Measuring in Reinforcement Learning With Delayed Negative Effects16 Oct 2025
- 2025Generalized Linear Markov Decision Process1 Jun 2025
- 2025Reinforcement Learning with Segment Feedback3 Feb 2025
Similar, not cited
Close in wording, no citation either way
- 2024Reinforcement Learning from Bagged Reward
- 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning
- 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
- 2024Rethinking State Disentanglement in Causal Reinforcement Learning
- 2026Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochs
- 2026Online Markov Decision Processes with Terminal Law Constraints
- 2026Temporal-Causal Inference for Reinforcement Learning via Automata Learning
- 2025Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environment