# Cross-fitted Proximal Learning for Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2604.05185 · HTML: https://mlsanity.com/paper/2604.05185/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 13 of 13)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Identifying Causal Effects With Proxy Variables of an Unmeasured Confounder](https://mlsanity.com/paper/1609.08816/) (2016)
- [Kernel Instrumental Variable Regression](https://mlsanity.com/paper/1906.00232/) (2019)
- [Deep Generalized Method of Moments for Instrumental Variable Analysis](https://mlsanity.com/paper/1905.12495/) (2019)
- [Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction](https://mlsanity.com/paper/2105.04544/) (2021)
- [Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2110.15332/) (2021)
- [A confounding bridge approach for double negative control inference on causal effects](https://mlsanity.com/paper/1808.04945/) (2018)
- [Connected and Automated Vehicles in Mixed-Traffic: Learning Human Driver Behavior for Effective On-Ramp Merging](https://mlsanity.com/paper/2304.00397/) (2023)
- [Learning Decision Policies with Instrumental Variables through Double Machine Learning](https://mlsanity.com/paper/2405.08498/) (2024)
- [A Framework for Effective AI Recommendations in Cyber-Physical-Human Systems](https://mlsanity.com/paper/2403.05715/) (2024)
- [Route Recommendations for Traffic Management Under Learned Partial Driver Compliance](https://mlsanity.com/paper/2504.02993/) (2025)
- [Off-Policy Evaluation for Sequential Persuasion Process with Unobserved Confounding](https://mlsanity.com/paper/2504.01211/) (2025)

## Abstract (arXiv)

Model-based reinforcement learning is attractive for sequential decision-making because it explicitly estimates reward and transition models and then supports planning through simulated rollouts. In offline settings with hidden confounding, however, models learned directly from observational data may be biased. This challenge is especially pronounced in partially observable systems, where latent factors may jointly affect actions, rewards, and future observations. Recent work has shown that policy evaluation in such confounded partially observable Markov decision processes (POMDPs) can be reduced to estimating reward-emission and observation-transition bridge functions satisfying conditional moment restrictions (CMRs). In this paper, we study the statistical estimation of these bridge functions. We formulate bridge learning as a CMR problem with nuisance objects given by a conditional mean embedding and a conditional density. We then develop a K-fold cross-fitted extension of the existing two-stage bridge estimator. The proposed procedure preserves the original bridge-based identification strategy while using the available data more efficiently than a single sample split. We also derive an oracle-comparator bound for the cross-fitted estimator and decompose the resulting error into a Stage I term induced by nuisance estimation and a Stage II term induced by empirical averaging.