mlsanityData to 14 Oct 2026

A Policy Gradient Method for Confounded POMDPs

arXiv 2305.17083 ↗First posted 26 May 2023stat.ML cs.LG econ.EMcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a policy gradient method for confounded partially observable Markov decision processes (POMDPs) with continuous state and observation spaces in the offline setting. We first establish a novel identification result to non-parametrically estimate any history-dependent policy gradient under POMDPs using the offline data. The identification enables us to solve a sequence of conditional moment restrictions and adopt the min-max learning procedure with general function approximation for estimating the policy gradient. We then provide a finite-sample non-asymptotic bound for estimating the gradient uniformly over a pre-specified policy class in terms of the sample size, length of horizon, concentratability coefficient and the measure of ill-posedness in solving the conditional moment restrictions. Lastly, by deploying the proposed gradient estimation in the gradient ascent algorithm, we show the global convergence of the proposed algorithm in finding the history-dependent optimal policy under some technical conditions. To the best of our knowledge, this is the first work studying the policy gradient method for POMDPs under the offline setting.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 14 of its 64 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

14 of 64 references.

References · 14

  1. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  2. 2019Orthogonal Statistical Learningcited by 151
  3. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  4. 2020An Introduction to Proximal Causal Learningcited by 95
  5. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  6. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  7. 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
  8. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  9. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  10. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  11. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 13
  12. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
2 more
  1. 2018Policy Gradient in Partially Observable Environments: Approximation and Convergencecited by 2
  2. 2023Personalized Pricing with Invalid Instrumental Variables: Identification, Estimation, and Policy Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 10
  2. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  3. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  4. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulatorscited by 1
  5. 2025Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPscited by 0
  6. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0