mlsanityData to 14 Oct 2026

Reinforcement Learning with Continuous Actions Under Unmeasured Confounding

arXiv 2505.00304 ↗First posted 1 May 2025stat.ML cs.LG stat.MEcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper addresses the challenge of offline policy learning in reinforcement learning with continuous action spaces when unmeasured confounders are present. While most existing research focuses on policy evaluation within partially observable Markov decision processes (POMDPs) and assumes discrete action spaces, we advance this field by establishing a novel identification result to enable the nonparametric estimation of policy value for a given target policy under an infinite-horizon framework. Leveraging this identification, we develop a minimax estimator and introduce a policy-gradient-based algorithm to identify the in-class optimal policy that maximizes the estimated policy value. Furthermore, we provide theoretical results regarding the consistency, finite-sample error bound, and regret bound of the resulting optimal policy. Extensive simulations and a real-world application using the German Family Panel data demonstrate the effectiveness of our proposed methodology.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 54 of its 81 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

54 of 81 references.

References · 54

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  7. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
38 more
  1. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  2. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  3. 2020An Introduction to Proximal Causal Learningcited by 95
  4. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  5. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  6. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  7. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  8. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  9. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  10. 2019Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Drivingcited by 59
  11. 2020Semiparametric proximal causal inferencecited by 59
  12. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
  13. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  14. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
  15. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  16. 2022Causal Imitation Learning with Unobserved Confounderscited by 41
  17. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  18. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  19. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  20. 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
  21. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  22. 2015Randomized sketches for kernels: Fast and optimal non-parametric regressioncited by 31
  23. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  24. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  25. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  26. 2020Statistically Efficient Off-Policy Policy Gradientscited by 25
  27. 2019A semiparametric instrumental variable approach to optimal treatment regimes under endogeneitycited by 22
  28. 2021Proximal Learning for Individualized Treatment Regimes Under Unmeasured Confoundingcited by 21
  29. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  30. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  31. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  32. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  33. 2022Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiencycited by 16
  34. 2019Sensitivity analysis via the proportion of unmeasured confoundingcited by 13
  35. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  36. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 10
  37. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  38. 2023A Policy Gradient Method for Confounded POMDPscited by 6

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  2. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0
  3. 2026Weighted Spline-Expanded Networks with Distributional Balancing for Continuous Treatment Effectscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Model-Based Reinforcement Learning Under Confounding
  2. 2022Offline Policy Evaluation and Optimization under Confounding
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  4. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs
  5. 2021Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
  6. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
  7. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPs
  8. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces