Data to 7 Oct 2026

Batch Policy Learning in Average Reward Markov Decision Processes

arXiv 2007.11771 ↗First posted 23 Jul 2020math.ST stat.ML stat.THcited by 69

Not on the map. Why?

Cited by 23 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 10; 2022: 17; 2023: 19; 2024: 11; 2025: 8; 2026: 3.

69 papers.

Sort:

By area:

2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
2022Jump-Start Reinforcement Learningcited by 102
2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24

38 more

2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
2022Conformal Off-policy Predictioncited by 14
2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
2025Doubly Robust Alignment for Large Language Modelscited by 11
2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
2021False Correlation Reduction for Offline Reinforcement Learningcited by 9
2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
2023Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharingcited by 7
2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 7
2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
2023Robust Multi-Agent Reinforcement Learning by Mutual Information Regularizationcited by 6
2024Regularized DeepIV with Model Selectioncited by 6
2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
2024On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processescited by 6
2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrabilitycited by 5
2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
2023ACPO: A Policy Optimization Algorithm for Average MDPs with Constraintscited by 4
2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
2021Pessimistic Model Selection for Offline Deep Reinforcement Learningcited by 3
2022Doubly Inhomogeneous Reinforcement Learningcited by 3

50 most cited of 69 All 69 (JSON)