# Towards Robust Off-policy Learning for Runtime Uncertainty

arXiv: https://arxiv.org/abs/2202.13337 · HTML: https://mlsanity.com/paper/2202.13337/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 27 Feb 2022.

## Builds on (most cited first, 22 of 22)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [Error-Bounded Correction of Noisy Labels](https://mlsanity.com/paper/2011.10077/) (2020)
- [Counterfactual Risk Minimization: Learning from Logged Bandit Feedback](https://mlsanity.com/paper/1502.02362/) (2015)
- [Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap](https://mlsanity.com/paper/1711.11286/) (2017)
- [Distributionally Robust Batch Contextual Bandits](https://mlsanity.com/paper/2006.05630/) (2020)
- [Policy Learning with Observational Data](https://mlsanity.com/paper/1702.02896/) (2017)
- [Imitation-Regularized Offline Learning](https://mlsanity.com/paper/1901.04723/) (2019)
- [Binomial Mixture Model With U-shape Constraint](https://mlsanity.com/paper/2107.13756/) (2021)

## Built on by (most cited first, 1 of 1)

- [Uncertainty-Aware Instance Reweighting for Off-Policy Learning](https://mlsanity.com/paper/2303.06389/) (2023)

## Abstract (arXiv)

Off-policy learning plays a pivotal role in optimizing and evaluating policies prior to the online deployment. However, during the real-time serving, we observe varieties of interventions and constraints that cause inconsistency between the online and offline settings, which we summarize and term as runtime uncertainty. Such uncertainty cannot be learned from the logged data due to its abnormality and rareness nature. To assert a certain level of robustness, we perturb the off-policy estimators along an adversarial direction in view of the runtime uncertainty. It allows the resulting estimators to be robust not only to observed but also unexpected runtime uncertainties. Leveraging this idea, we bring runtime-uncertainty robustness to three major off-policy learning methods: the inverse propensity score method, reward-model method, and doubly robust method. We theoretically justify the robustness of our methods to runtime uncertainty, and demonstrate their effectiveness using both the simulation and the real-world online experiments.