mlsanityData to 14 Oct 2026

Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learning

arXiv 2412.05783 ↗First posted 8 Dec 2024cs.LG stat.MLcited by 5

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper studies off-policy evaluation (OPE) in the presence of unmeasured confounders. Inspired by the two-way fixed effects regression model widely used in the panel data literature, we propose a two-way unmeasured confounding assumption to model the system dynamics in causal reinforcement learning and develop a two-way deconfounder algorithm that devises a neural tensor network to simultaneously learn both the unmeasured confounders and the system dynamics, based on which a model-based estimator can be constructed for consistent policy value estimation. We illustrate the effectiveness of the proposed estimator through theoretical results and numerical experiments.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 71 of its 109 references that are in the map. First posted on arXiv 8 Dec 2024.

Builds on

71 of 109 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning15 Apr

Other references · 70

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  3. 2017Neural Collaborative Filteringcited by 1,111
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  7. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  8. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 410
  9. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  12. 2015A Review of Relational Machine Learning for Knowledge Graphscited by 265
38 more
  1. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  2. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  3. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  4. 2017A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Networkcited by 143
  5. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  6. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  7. 2018The Blessings of Multiple Causescited by 104
  8. 2020An Introduction to Proximal Causal Learningcited by 95
  9. 2019Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounderscited by 75
  10. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  11. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  12. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  13. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  14. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  15. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  16. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
  17. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  18. 2021Universal Off-Policy Evaluationcited by 43
  19. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  20. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  21. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  22. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  23. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  24. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  25. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  26. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  27. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  28. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  29. 2019Using Embeddings to Correct for Unobserved Confounding in Networkscited by 31
  30. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  31. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  32. 2021A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Modelscited by 29
  33. 2019Importance Resampling for Off-policy Predictioncited by 26
  34. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  35. 2021Sequential Deconfounding for Causal Inference with Unobserved Confounderscited by 24
  36. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  37. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  38. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  2. 2025Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approachcited by 1
  3. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  4. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0
  5. 2026A Closer Look at the Application of Causal Inference in Graph Representation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Graphical Approach to State Variable Selection in Off-policy Learning
  2. 2022Causal Deep Reinforcement Learning Using Observational Data
  3. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning
  4. 2025DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
  5. 2022Offline Reinforcement Learning with Causal Structured World Models
  6. 2024Skill or Luck? Return Decomposition via Advantage Functions
  7. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
  8. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs