# Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings

arXiv: https://arxiv.org/abs/2605.14284 · HTML: https://mlsanity.com/paper/2605.14284/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 77 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 30 of 31)

- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations](https://mlsanity.com/paper/2002.04083/) (2020)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation](https://mlsanity.com/paper/2002.09516/) (2020)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/1908.08526/) (2019)
- [FALKON: An Optimal Large Scale Kernel Method](https://mlsanity.com/paper/1705.10958/) (2017)
- [Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations](https://mlsanity.com/paper/2206.08311/) (2022)
- [Fast Two-Sample Testing with Analytic Representations of Probability Measures](https://mlsanity.com/paper/1506.04725/) (2015)
- [Importance Sampling Policy Evaluation with an Estimated Behavior Policy](https://mlsanity.com/paper/1806.01347/) (2018)
- [Estimating average causal effects from patient trajectories](https://mlsanity.com/paper/2203.01228/) (2022)
- [Importance Resampling for Off-policy Prediction](https://mlsanity.com/paper/1906.04328/) (2019)
- [A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation](https://mlsanity.com/paper/2106.06854/) (2021)
- [Model-agnostic meta-learners for estimating heterogeneous treatment effects over time](https://mlsanity.com/paper/2407.05287/) (2024)
- [More Efficient Off-Policy Evaluation through Regularized Targeted Learning](https://mlsanity.com/paper/1912.06292/) (2019)
- [Non-parametric causal effects based on longitudinal modified treatment policies](https://mlsanity.com/paper/2006.01366/) (2020)
- [Longitudinal Targeted Minimum Loss-based Estimation with Temporal-Difference Heterogeneous Transformer](https://mlsanity.com/paper/2404.04399/) (2024)
- [IGC-Net for conditional average potential outcome estimation over time](https://mlsanity.com/paper/2405.21012/) (2024)
- [Low Variance Off-policy Evaluation with State-based Importance Sampling](https://mlsanity.com/paper/2212.03932/) (2022)
- [Kernel Debiased Plug-in Estimation: Simultaneous, Automated Debiasing without Influence Functions for Many Target Parameters](https://mlsanity.com/paper/2306.08598/) (2023)
- [GST-UNet: A Neural Framework for Spatiotemporal Causal Inference with Time-Varying Confounding](https://mlsanity.com/paper/2502.05295/) (2025)
- [State Relevance for Off-Policy Evaluation](https://mlsanity.com/paper/2109.06310/) (2021)
- [Deep Learning Methods for the Noniterative Conditional Expectation G-Formula for Causal Inference from Complex Observational Data](https://mlsanity.com/paper/2410.21531/) (2024)
- [Estimating the treatment effect over time under general interference through deep learner integrated TMLE](https://mlsanity.com/paper/2412.04799/) (2024)

## Abstract (arXiv)

Comparative evaluation of multiple dynamic treatment policies is essential for healthcare and policy decisions, yet conventional longitudinal causal inference methods estimate each in isolation, preventing information sharing across counterfactuals. We demonstrate that this separate estimation paradigm induces a structurally uncontrolled second-order bias, inflating finite-sample variance even after standard debiasing with longitudinal targeted maximum likelihood estimation(LTMLE). To address this, we propose a policy-aware reparameterization of Iterative Conditional Expectation (ICE) Q-functions that enables joint estimation through shared representations. We implement this approach in the Policy-Encoded Q Network (PEQ-Net), an architecture centered on a shared policy encoder. The encoder is trained using kernel mean embeddings, ensuring that the learned representation space reflects population-level policy dissimilarities. After applying an LTMLE correction step, we prove this design imposes a structural constraint on the second-order remainder, thereby stabilizing finite-sample variance. Experiments on semi-synthetic datasets demonstrate that PEQ-Net consistently outperforms existing ICE-based methods, achieving substantial reductions in root-mean-square error, particularly when evaluating closely related policies.