Off-policy Evaluation with Deeply-abstracted States
Not on the map. Why?
Abstract (arXiv)
Off-policy evaluation (OPE) is crucial for assessing a target policy's impact offline before its deployment. However, achieving accurate OPE in large state spaces remains challenging. This paper studies state abstractions -- originally designed for policy learning -- in the context of OPE. Our contributions are three-fold: (i) We define a set of irrelevance conditions central to learning state abstractions for OPE, and derive a backward-model-irrelevance condition for achieving irrelevance in %sequential and (marginalized) importance sampling ratios by constructing a time-reversed Markov decision process (MDP). (ii) We propose a novel iterative procedure that sequentially projects the original state space into a smaller space, resulting in a deeply-abstracted state, which substantially simplifies the sample complexity of OPE arising from high cardinality. (iii) We prove the Fisher consistencies of various OPE estimators when applied to our proposed abstract state spaces.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 63 of its 114 references that are in the map. First posted on arXiv 27 Jun 2024.
Builds on
63 of 114 references.
References · 63
- 2015Trust Region Policy Optimizationcited by 2,503
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
- 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
- 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
- 2019Batch Policy Learning under Constraintscited by 242
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
- 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
- 2019A Theoretical Analysis of Deep Q-Learningcited by 212
38 more
- 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
- 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
- 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
- 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
- 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
- 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
- 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
- 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
- 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
- 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
- 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
- 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
- 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
- 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
- 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
- 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
- 2017Near Optimal Behavior via Approximate State Abstractioncited by 63
- 2018Deep Neural Networks for Estimation and Inferencecited by 62
- 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
- 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
- 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
- 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
- 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
- 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
- 2018Combined Reinforcement Learning via Abstract Representationscited by 37
- 2020Design and Analysis of Switchback Experimentscited by 37
- 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
- 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
- 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
- 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
- 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
- 2022A Theory of Abstraction in Reinforcement Learningcited by 26
- 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
- 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
- 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
- 2022Switchback Experiments under Geometric Mixingcited by 17
- 2021Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approachcited by 16
- 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.