Data to 14 Oct 2026

Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation

arXiv 1810.12429 ↗First posted 29 Oct 2018cs.LG cs.AI cs.SYcited by 227

Not on the map. Why?

Cited by 60 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon problems. In the extreme case of in infinite-horizon problems, the variance of an IS-based estimator may even be unbounded. In this paper, we propose a new off-policy estimation method that applies IS directly on the stationary state-visitation distributions to avoid the exploding variance issue faced by existing estimators.Our key contribution is a novel approach to estimating the density ratio of two stationary distributions, with trajectories sampled from only the behavior distribution. We develop a mini-max loss function for the estimation problem, and derive a closed-form solution for the case of RKHS. We support our method with both theoretical and empirical analyses.

Key facts. Cited by 227 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 8; 2021: 49; 2022: 46; 2023: 43; 2024: 26; 2025: 26; 2026: 28.

227 papers.

Sort:

By area:

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2019Behavior Regularized Offline Reinforcement Learningcited by 662
2020Is Pessimism Provably Efficient for Offline RL?cited by 364
2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
2021Conservative Offline Distributional Reinforcement Learningcited by 79
2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56

38 more

2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
2021Universal Off-Policy Evaluationcited by 43
2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 31
2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
2021Fast Rates for the Regret of Offline Reinforcement Learningcited by 26
2021Active Offline Policy Selectioncited by 26
2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
2024Diffusion Model for Data-Driven Black-Box Optimizationcited by 24
2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 22
2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
2022Markovian Interference in Experimentscited by 22
2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 21
2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 20
2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 18

50 most cited of 227 All 227 (JSON)