mlsanityData to 7 Oct 2026

Average-Reward Off-Policy Policy Evaluation with Function Approximation

arXiv 2101.02808 ↗First posted 8 Jan 2021cs.LG cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function. For this problem, bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad (Sutton & Barto, 2018). To address the deadly triad, we propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting. In terms of estimating the differential value function, the algorithms are the first convergent off-policy linear function approximation algorithms. In terms of estimating the reward rate, the algorithms are the first convergent off-policy linear function approximation algorithms that do not require estimating the density ratio. We demonstrate empirically the advantage of the proposed algorithms, as well as their nonlinear variants, over a competitive density-ratio-based approach, in a simple domain as well as challenging robot simulation tasks.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 8 Jan 2021.

Builds on

32 of 59 references.

References · 32

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2017Deep Reinforcement Learning that Matterscited by 727
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  7. 2019Challenges of Real-World Reinforcement Learningcited by 282
  8. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  9. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  10. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  11. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  12. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
20 more
  1. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  2. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  3. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  4. 2017Stochastic Variance Reduction Methods for Policy Evaluationcited by 56
  5. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  6. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  7. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  8. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  9. 2020Finite-Sample Analysis of Proximal Gradient TD Algorithmscited by 32
  10. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  11. 2018Representation Balancing MDPs for Off-Policy Policy Evaluationcited by 20
  12. 2014Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spacescited by 19
  13. 2013Distributed Policy Evaluation Under Multiple Behavior Strategiescited by 17
  14. 2019Combining Parametric and Nonparametric Models for Off-Policy Evaluationcited by 17
  15. 2020Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learningcited by 16
  16. 2019Planning with Expectation Modelscited by 12
  17. 2017On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learningcited by 11
  18. 2018A Block Coordinate Ascent Algorithm for Mean-Variance Optimizationcited by 11
  19. 2020A maximum-entropy approach to off-policy evaluation in average-reward MDPscited by 9
  20. 2019A Convergent Off-Policy Temporal Difference Algorithmcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 8; 2024: 5; 2025: 6; 2026: 5.

30 papers.

Sort:

By area:

  1. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  2. 2022Stochastic first-order methods for average-reward Markov decision processescited by 30
  3. 2022Markovian Interference in Experimentscited by 22
  4. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  5. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  6. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14
  7. 2023Finite-Time Analysis of Whittle Index based Q-Learning for Restless Multi-Armed Bandits with Neural Network Function Approximationcited by 14
  8. 2021Average-Reward Learning and Planning with Optionscited by 8
  9. 2023Revisiting a Design Choice in Gradient Temporal Difference Learningcited by 7
  10. 2023Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithmscited by 6
  11. 2023Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learningcited by 5
  12. 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 5
18 more
  1. 2024Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  2. 2025Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  3. 2025Towards Formalizing Reinforcement Learning Theory: A Robbins-Siegmund Approachcited by 5
  4. 2023ACPO: A Policy Optimization Algorithm for Average MDPs with Constraintscited by 4
  5. 2024Transformable Gaussian Reward Function for Socially-Aware Navigation with Deep Reinforcement Learningcited by 3
  6. 2025Average Reward Reinforcement Learning for Wireless Radio Resource Managementcited by 3
  7. 2023A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothnesscited by 2
  8. 2024e-COP : Episodic Constrained Optimization of Policiescited by 2
  9. 2024Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processescited by 2
  10. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  11. 2025Implicit Updates for Average-Reward Temporal Difference Learningcited by 2
  12. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processescited by 2
  13. 2026Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotientscited by 1
  14. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  15. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  16. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  17. 2026Bridging the Gap Between Average and Discounted TD Learningcited by 0
  18. 2026Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Driftcited by 0