Cross-fitted Proximal Learning for Model-Based Reinforcement Learning
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Model-based reinforcement learning is attractive for sequential decision-making because it explicitly estimates reward and transition models and then supports planning through simulated rollouts. In offline settings with hidden confounding, however, models learned directly from observational data may be biased. This challenge is especially pronounced in partially observable systems, where latent factors may jointly affect actions, rewards, and future observations. Recent work has shown that policy evaluation in such confounded partially observable Markov decision processes (POMDPs) can be reduced to estimating reward-emission and observation-transition bridge functions satisfying conditional moment restrictions (CMRs). In this paper, we study the statistical estimation of these bridge functions. We formulate bridge learning as a CMR problem with nuisance objects given by a conditional mean embedding and a conditional density. We then develop a K-fold cross-fitted extension of the existing two-stage bridge estimator. The proposed procedure preserves the original bridge-based identification strategy while using the available data more efficiently than a single sample split. We also derive an oracle-comparator bound for the cross-fitted estimator and decompose the resulting error into a Stage I term induced by nuisance estimation and a Stage II term induced by empirical averaging.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 26 references that are in the map. First posted on arXiv 6 Apr 2026.
Builds on
13 of 26 references.
References · 13
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 364
- 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
- 2019Kernel Instrumental Variable Regressioncited by 118
- 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
- 2021Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restrictioncited by 55
- 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
- 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
- 2023Connected and Automated Vehicles in Mixed-Traffic: Learning Human Driver Behavior for Effective On-Ramp Mergingcited by 7
- 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
- 2024A Framework for Effective AI Recommendations in Cyber-Physical-Human Systemscited by 4
- 2025Route Recommendations for Traffic Management Under Learned Partial Driver Compliancecited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Model-Based Reinforcement Learning Under Confounding
- 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
- 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
- 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
- 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders
- 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
- 2024Rethinking State Disentanglement in Causal Reinforcement Learning
- 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs