# Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inference

arXiv: https://arxiv.org/abs/2501.06926 · HTML: https://mlsanity.com/paper/2501.06926/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 12 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Built on by (most cited first, 7 of 7)

- [Calibeating Prediction-Powered Inference](https://mlsanity.com/paper/2604.21260/) (2026)
- [Bellman Calibration for V-Learning in Offline Reinforcement Learning](https://mlsanity.com/paper/2512.23694/) (2025)
- [Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2512.24407/) (2025)
- [Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting](https://mlsanity.com/paper/2512.23805/) (2025)
- [Fitted Occupancy-Ratio Evaluation without Bellman Completeness](https://mlsanity.com/paper/2607.05375/) (2026)
- [An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes](https://mlsanity.com/paper/2509.26429/) (2025)
- [Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning](https://mlsanity.com/paper/2608.24858/) (2026)

## Abstract (arXiv)

Double reinforcement learning (DRL) provides efficient off-policy inference for policy values in nonparametric Markov decision processes (MDPs), but fully nonparametric estimators can be unstable when intertemporal overlap is weak and occupancy ratios are high-dimensional. This limitation is especially relevant for long-term causal inference from randomized experiments: randomization ensures overlap in treatment assignment, but not over future state trajectories induced by continued intervention use. We develop semiparametric DRL for continuous linear functionals of the infinite-horizon Q-function. Rather than impose linear MDP structure on the reward and transition laws, we place working semiparametric restrictions on the Q-function itself, the solution of the discounted Bellman equation. When correct, these restrictions can improve efficiency relative to unrestricted DRL while allowing rich, possibly infinite-dimensional models. To avoid relying on correct specification, we define the estimand through weighted Bellman-residual minimization. The resulting projection target remains meaningful under misspecification and recovers the original functional under correct specification. For this class of parameters, we derive efficient influence functions and efficiency bounds, construct model-robust automatically debiased estimators, and develop minimax criteria for estimating the Q- and Riesz functions. Under correct specification, optimally weighted versions attain the semiparametric efficiency bound in the restricted model.