mlsanityData to 7 Oct 2026

Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data

arXiv 2403.11841 ↗First posted 18 Mar 2024stat.ML cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In real-world scenarios, datasets collected from randomized experiments are often constrained by size, due to limitations in time and budget. As a result, leveraging large observational datasets becomes a more attractive option for achieving high-quality policy learning. However, most existing offline reinforcement learning (RL) methods depend on two key assumptions--unconfoundedness and positivity--which frequently do not hold in observational data contexts. Recognizing these challenges, we propose a novel policy learning algorithm, PESsimistic CAusal Learning (PESCAL). We utilize the mediator variable based on front-door criterion to remove the confounding bias; additionally, we adopt the pessimistic principle to address the distributional shift between the action distributions induced by candidate policies, and the behavior policy that generates the observational data. Our key observation is that, by incorporating auxiliary variables that mediate the effect of actions on system dynamics, it is sufficient to learn a lower bound of the mediator distribution function, instead of the Q-function, to partially mitigate the issue of distributional shift. This insight significantly simplifies our algorithm, by circumventing the challenging task of sequential uncertainty quantification for the estimated Q-function. Moreover, we provide theoretical guarantees for the algorithms we propose, and demonstrate their efficacy through simulations, as well as real-world experiments utilizing offline datasets from a leading ride-hailing platform.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 82 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

33 of 82 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  8. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  11. 2018Confounding-Robust Policy Improvementcited by 73
  12. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
21 more
  1. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  2. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  3. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  4. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  5. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36
  6. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  7. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  8. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  9. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  10. 2020On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spacescited by 19
  11. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  12. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  13. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  14. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  15. 2022Model-Based Reinforcement Learning for Offline Zero-Sum Markov Gamescited by 11
  16. 2021Asymptotic Theory for IV-Based Reinforcement Learning with Potential Endogeneitycited by 8
  17. 2022Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approachcited by 7
  18. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  19. 2021Stateful Offline Contextual Policy Evaluation and Learningcited by 6
  20. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  21. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Semi-pessimistic Reinforcement Learning
  2. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  3. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  4. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  5. 2026Pessimistic Auxiliary Policy for Offline Reinforcement Learning
  6. 2026Generalization in offline RL: The structure is more important than the amount of pessimism
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning
  8. 2025POLAR: A Pessimistic Model-based Policy Learning Algorithm for Dynamic Treatment Regimes