# Off-Policy Risk Assessment in Contextual Bandits

arXiv: https://arxiv.org/abs/2104.08977 · HTML: https://mlsanity.com/paper/2104.08977/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 81 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 21 of 21)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Doubly Robust Policy Evaluation and Optimization](https://mlsanity.com/paper/1503.02834/) (2015)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://mlsanity.com/paper/1911.01546/) (2019)
- [Adaptive Sampling for Stochastic Risk-Averse Learning](https://mlsanity.com/paper/1910.12511/) (2019)
- [Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control](https://mlsanity.com/paper/1506.02632/) (2015)
- [Learning Bounds for Risk-sensitive Learning](https://mlsanity.com/paper/2006.08138/) (2020)
- [Best Arm Identification for Contaminated Bandits](https://mlsanity.com/paper/1802.09514/) (2018)
- [Distribution oblivious, risk-aware algorithms for multi-armed bandits with unbounded rewards](https://mlsanity.com/paper/1906.00569/) (2019)
- [Statistical Learning with Conditional Value at Risk](https://mlsanity.com/paper/2002.05826/) (2020)
- [Generalized Risk-Aversion in Stochastic Multi-Armed Bandits](https://mlsanity.com/paper/1405.0833/) (2014)
- [Risk-Constrained Thompson Sampling for CVaR Bandits](https://mlsanity.com/paper/2011.08046/) (2020)
- [High-Confidence Off-Policy (or Counterfactual) Variance Estimation](https://mlsanity.com/paper/2101.09847/) (2021)
- [X-Armed Bandits: Optimizing Quantiles, CVaR and Other Risks](https://mlsanity.com/paper/1904.08205/) (2019)

## Built on by (most cited first, 29 of 29)

- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Distributional Offline Policy Evaluation with Predictive Error Guarantees](https://mlsanity.com/paper/2302.09456/) (2023)
- [A Unifying Theory of Thompson Sampling for Continuous Risk-Averse Bandits](https://mlsanity.com/paper/2108.11345/) (2021)
- [A Survey of Risk-Aware Multi-Armed Bandits](https://mlsanity.com/paper/2205.05843/) (2022)
- [Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures](https://mlsanity.com/paper/2306.02399/) (2023)
- [Tipping Point Forecasting in Non-Stationary Dynamics on Function Spaces](https://mlsanity.com/paper/2308.08794/) (2023)
- [Off-Policy Risk Assessment in Markov Decision Processes](https://mlsanity.com/paper/2209.10444/) (2022)
- [Supervised Learning with General Risk Functionals](https://mlsanity.com/paper/2206.13648/) (2022)
- [Risk Verification of Stochastic Systems with Neural Network Controllers](https://mlsanity.com/paper/2209.09881/) (2022)
- [SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation](https://mlsanity.com/paper/2311.18206/) (2023)
- [ESCADA: Efficient Safety and Context Aware Dose Allocation for Precision Medicine](https://mlsanity.com/paper/2111.13415/) (2021)
- [Quantile Off-Policy Evaluation via Deep Conditional Generative Learning](https://mlsanity.com/paper/2212.14466/) (2022)
- [Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effect](https://mlsanity.com/paper/2411.14341/) (2024)
- [Off-Policy Evaluation with Out-of-Sample Guarantees](https://mlsanity.com/paper/2301.08649/) (2023)
- [Probabilistic Offline Policy Ranking with Approximate Bayesian Computation](https://mlsanity.com/paper/2312.11551/) (2023)
- [Risk-sensitive Bandits: Arm Mixture Optimality and Regret-efficient Algorithms](https://mlsanity.com/paper/2503.08896/) (2025)
- [Doubly-Robust Estimation of Counterfactual Policy Mean Embeddings](https://mlsanity.com/paper/2506.02793/) (2025)
- [Conditionally Risk-Averse Contextual Bandits](https://mlsanity.com/paper/2210.13573/) (2022)
- [Time-uniform confidence bands for the CDF under nonstationarity](https://mlsanity.com/paper/2302.14248/) (2023)
- [A Distribution Optimization Framework for Confidence Bounds of Risk Measures](https://mlsanity.com/paper/2306.07059/) (2023)
- [Distributional Off-Policy Evaluation for Slate Recommendations](https://mlsanity.com/paper/2308.14165/) (2023)
- [Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effect](https://mlsanity.com/paper/2502.04673/) (2025)
- [Kernel Treatment Effects with Adaptively Collected Data](https://mlsanity.com/paper/2510.10245/) (2025)
- [Functional Linear Regression of Cumulative Distribution Functions](https://mlsanity.com/paper/2205.14545/) (2022)
- [Bayesian Counterfactual Mean Embeddings and Off-Policy Evaluation](https://mlsanity.com/paper/2211.01518/) (2022)
- [Externally Valid Policy Evaluation Combining Trial and Observational Data](https://mlsanity.com/paper/2310.14763/) (2023)
- [Pessimistic Risk-Aware Policy Learning in Contextual Bandits](https://mlsanity.com/paper/2605.15620/) (2026)

## Abstract (arXiv)

Even when unable to run experiments, practitioners can evaluate prospective policies, using previously logged data. However, while the bandits literature has adopted a diverse set of objectives, most research on off-policy evaluation to date focuses on the expected reward. In this paper, we introduce Lipschitz risk functionals, a broad class of objectives that subsumes conditional value-at-risk (CVaR), variance, mean-variance, many distorted risks, and CPT risks, among others. We propose Off-Policy Risk Assessment (OPRA), a framework that first estimates a target policy's CDF and then generates plugin estimates for any collection of Lipschitz risks, providing finite sample guarantees that hold simultaneously over the entire class. We instantiate OPRA with both importance sampling and doubly robust estimators. Our primary theoretical contributions are (i) the first uniform concentration inequalities for both CDF estimators in contextual bandits and (ii) error bounds on our Lipschitz risk estimates, which all converge at a rate of O(1/√n).