Off-Policy Evaluation for Action-Dependent Non-Stationary Environments
Not on the map. Why?
Abstract (arXiv)
Methods for sequential decision-making are often built upon a foundational assumption that the underlying decision process is stationary. This limits the application of such methods because real-world problems are often subject to changes due to external factors (passive non-stationarity), changes induced by interactions with the system itself (active non-stationarity), or both (hybrid non-stationarity). In this work, we take the first steps towards the fundamental challenge of on-policy and off-policy evaluation amidst structured changes due to active, passive, or hybrid non-stationarity. Towards this goal, we make a higher-order stationarity assumption such that non-stationarity results in changes over time, but the way changes happen is fixed. We propose, OPEN, an algorithm that uses a double application of counterfactual reasoning and a novel importance-weighted instrument-variable regression to obtain both a lower bias and a lower variance estimate of the structure in the changes of a policy's past performances. Finally, we show promising results on how OPEN can be used to predict future performances for several domains inspired by real-world applications that exhibit non-stationarity.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 52 of its 119 references that are in the map. First posted on arXiv 24 Jan 2023.
Builds on
52 of 119 references.
References · 52
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
- 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
- 2019Challenges of Real-World Reinforcement Learningcited by 282
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
- 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
- 2017Learning with Opponent-Learning Awarenesscited by 218
- 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
- 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
- 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
38 more
- 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
- 2020Learning Latent Representations to Influence Multi-Agent Interactioncited by 92
- 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
- 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
- 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
- 2019Weighted Linear Bandits for Non-Stationary Environmentscited by 84
- 2019Influence-Based Multi-Agent Explorationcited by 75
- 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
- 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
- 2020Learning Deep Features in Instrumental Variable Regressioncited by 65
- 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
- 2020A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environmentscited by 62
- 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
- 2017Rotting Banditscited by 57
- 2016Q(λ) with Off-Policy Correctionscited by 48
- 2021Universal Off-Policy Evaluationcited by 43
- 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
- 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
- 2020Optimizing for the Future in Non-Stationary MDPscited by 37
- 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
- 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
- 2021Influencing Towards Stable Multi-Agent Interactionscited by 24
- 2020Nonstationary Reinforcement Learning with Linear Function Approximationcited by 23
- 2018Me, My Echo Chamber, and I: Introspection on Social Media Polarizationcited by 20
- 2013Nash Convergence of Gradient Dynamics in Iterated General-Sum Gamescited by 18
- 2021Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detectioncited by 18
- 2015Emphatic Temporal-Difference Learningcited by 16
- 2015Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regretcited by 15
- 2020On Deep Instrumental Variables Estimatecited by 14
- 2019Neural Replicator Dynamicscited by 13
- 2020Minimax Value Interval for Off-Policy Evaluation and Policy Optimizationcited by 11
- 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
- 2019Reinforcement Learning in Non-Stationary Environmentscited by 10
- 2020Reinforcement Learning for Strategic Recommendationscited by 8
- 2021Meta-Reinforcement Learning by Tracking Task Non-stationaritycited by 8
- 2019Cascading Non-Stationary Bandits: Online Learning to Rank in the Non-Stationary Cascade Modelcited by 5
- 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
- 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularizationcited by 5
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2024: 2.
2 papers.