Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iteration
Not on the map. Why?
Abstract (arXiv)
Policy gradient (PG) estimation becomes a challenge when we are not allowed to sample with the target policy but only have access to a dataset generated by some unknown behavior policy. Conventional methods for off-policy PG estimation often suffer from either significant bias or exponentially large variance. In this paper, we propose the double Fitted PG estimation (FPG) algorithm. FPG can work with an arbitrary policy parameterization, assuming access to a Bellman-complete value function class. In the case of linear value function approximation, we provide a tight finite-sample upper bound on policy gradient estimation error, that is governed by the amount of distribution mismatch measured in feature space. We also establish the asymptotic normality of FPG estimation error with a precise covariance characterization, which is further shown to be statistically optimal with a matching Cramer-Rao lower bound. Empirically, we evaluate the performance of FPG on both policy gradient estimation and policy optimization, using either softmax tabular or ReLU policy networks. Under various metrics, our results show that FPG significantly outperforms existing off-policy PG estimation methods based on importance sampling and variance reduction techniques.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 31 Jan 2022.
Builds on
26 of 42 references.
References · 26
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
- 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
- 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
- 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
- 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
- 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
- 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
- 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
- 2018Stochastic Variance-Reduced Policy Gradientcited by 103
14 more
- 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
- 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
- 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 58
- 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
- 2021Corruption-Robust Offline Reinforcement Learningcited by 51
- 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
- 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
- 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
- 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
- 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
- 2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
- 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
- 2013Policy Improvement for POMDPs Using Normalized Importance Samplingcited by 3
- 2020A Nonparametric Off-Policy Policy Gradientcited by 3
Built on by
Citing papers per year: 2022: 1.
1 paper.