# Batch Policy Learning in Average Reward Markov Decision Processes

arXiv: https://arxiv.org/abs/2007.11771 · HTML: https://mlsanity.com/paper/2007.11771/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2020.

## Built on by (most cited first, 30 of 69)

- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism](https://mlsanity.com/paper/2103.12021/) (2021)
- [Jump-Start Reinforcement Learning](https://mlsanity.com/paper/2204.02372/) (2022)
- [Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning](https://mlsanity.com/paper/2108.08812/) (2021)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Deeply-Debiased Off-Policy Interval Estimation](https://mlsanity.com/paper/2105.04646/) (2021)
- [A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes](https://mlsanity.com/paper/2111.06784/) (2021)
- [On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluation](https://mlsanity.com/paper/2201.06169/) (2022)
- [Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process](https://mlsanity.com/paper/2202.10589/) (2022)
- [Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian](https://mlsanity.com/paper/2211.00716/) (2022)
- [Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage](https://mlsanity.com/paper/2106.03207/) (2021)
- [Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability](https://mlsanity.com/paper/2110.12343/) (2021)
- [Future-Dependent Value-Based Off-Policy Evaluation in POMDPs](https://mlsanity.com/paper/2207.13081/) (2022)
- [Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic](https://mlsanity.com/paper/2603.01162/) (2026)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes](https://mlsanity.com/paper/2209.08666/) (2022)
- [Projected State-action Balancing Weights for Offline Reinforcement Learning](https://mlsanity.com/paper/2109.04640/) (2021)
- [Conformal Off-policy Prediction](https://mlsanity.com/paper/2206.06711/) (2022)
- [Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning](https://mlsanity.com/paper/2206.07166/) (2022)
- [Offline Multi-task Transfer RL with Representational Penalization](https://mlsanity.com/paper/2402.12570/) (2024)
- [Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward](https://mlsanity.com/paper/2206.06426/) (2022)
- [Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning](https://mlsanity.com/paper/2301.12714/) (2023)
- [Testing Stationarity and Change Point Detection in Reinforcement Learning](https://mlsanity.com/paper/2203.01707/) (2022)
- [Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage](https://mlsanity.com/paper/2310.18434/) (2023)
- [Doubly Robust Alignment for Large Language Models](https://mlsanity.com/paper/2506.01183/) (2025)
- [Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons](https://mlsanity.com/paper/2202.13163/) (2022)
- [False Correlation Reduction for Offline Reinforcement Learning](https://mlsanity.com/paper/2110.12468/) (2021)
- [On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes](https://mlsanity.com/paper/2408.16262/) (2024)
- [Policy Evaluation for Temporal and/or Spatial Dependent Experiments](https://mlsanity.com/paper/2202.10887/) (2022)
- [Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap](https://mlsanity.com/paper/2402.08201/) (2024)

## Abstract (arXiv)

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.