mlsanityData to 13 Oct 2026

Off-Policy Risk Assessment in Contextual Bandits

arXiv 2104.08977 ↗First posted 18 Apr 2021cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Even when unable to run experiments, practitioners can evaluate prospective policies, using previously logged data. However, while the bandits literature has adopted a diverse set of objectives, most research on off-policy evaluation to date focuses on the expected reward. In this paper, we introduce Lipschitz risk functionals, a broad class of objectives that subsumes conditional value-at-risk (CVaR), variance, mean-variance, many distorted risks, and CPT risks, among others. We propose Off-Policy Risk Assessment (OPRA), a framework that first estimates a target policy's CDF and then generates plugin estimates for any collection of Lipschitz risks, providing finite sample guarantees that hold simultaneously over the entire class. We instantiate OPRA with both importance sampling and doubly robust estimators. Our primary theoretical contributions are (i) the first uniform concentration inequalities for both CDF estimators in contextual bandits and (ii) error bounds on our Lipschitz risk estimates, which all converge at a rate of O(1/√n).

Key facts. Cited by 29 papers in the map (data through 13 Oct 2026). Builds on 21 of its 81 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

21 of 81 references.

References · 21

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 745
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 465
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 363
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  5. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  6. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  8. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  9. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  10. 2015Policy Gradient for Coherent Risk Measurescited by 75
  11. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  12. 2019Adaptive Sampling for Stochastic Risk-Averse Learningcited by 39
9 more
  1. 2015Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Controlcited by 34
  2. 2020Learning Bounds for Risk-sensitive Learningcited by 33
  3. 2018Best Arm Identification for Contaminated Banditscited by 23
  4. 2019Distribution oblivious, risk-aware algorithms for multi-armed bandits with unbounded rewardscited by 19
  5. 2020Statistical Learning with Conditional Value at Riskcited by 19
  6. 2014Generalized Risk-Aversion in Stochastic Multi-Armed Banditscited by 11
  7. 2020Risk-Constrained Thompson Sampling for CVaR Banditscited by 10
  8. 2021High-Confidence Off-Policy (or Counterfactual) Variance Estimationcited by 8
  9. 2019X-Armed Bandits: Optimizing Quantiles, CVaR and Other Riskscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 10; 2024: 1; 2025: 4; 2026: 1.

29 papers.

Sort:

By area:

  1. 2021Universal Off-Policy Evaluationcited by 43
  2. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  3. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  4. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 20
  5. 2021A Unifying Theory of Thompson Sampling for Continuous Risk-Averse Banditscited by 9
  6. 2022A Survey of Risk-Aware Multi-Armed Banditscited by 8
  7. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 8
  8. 2023Tipping Point Forecasting in Non-Stationary Dynamics on Function Spacescited by 8
  9. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  10. 2022Supervised Learning with General Risk Functionalscited by 6
  11. 2022Risk Verification of Stochastic Systems with Neural Network Controllerscited by 6
  12. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
17 more
  1. 2021ESCADA: Efficient Safety and Context Aware Dose Allocation for Precision Medicinecited by 5
  2. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  3. 2024Logarithmic Neyman Regret for Adaptive Estimation of the Average Treatment Effectcited by 4
  4. 2023Probabilistic Offline Policy Ranking with Approximate Bayesian Computationcited by 3
  5. 2025Risk-sensitive Bandits: Arm Mixture Optimality and Regret-efficient Algorithmscited by 3
  6. 2025Doubly-Robust Estimation of Counterfactual Policy Mean Embeddingscited by 3
  7. 2022Conditionally Risk-Averse Contextual Banditscited by 2
  8. 2023Off-Policy Evaluation with Out-of-Sample Guaranteescited by 2
  9. 2023Time-uniform confidence bands for the CDF under nonstationaritycited by 2
  10. 2023A Distribution Optimization Framework for Confidence Bounds of Risk Measurescited by 2
  11. 2023Distributional Off-Policy Evaluation for Slate Recommendationscited by 2
  12. 2025Optimistic Algorithms for Adaptive Estimation of the Average Treatment Effectcited by 2
  13. 2022Functional Linear Regression of Cumulative Distribution Functionscited by 1
  14. 2025Kernel Treatment Effects with Adaptively Collected Datacited by 1
  15. 2022Bayesian Counterfactual Mean Embeddings and Off-Policy Evaluationcited by 0
  16. 2023Externally Valid Policy Evaluation Combining Trial and Observational Datacited by 0
  17. 2026Pessimistic Risk-Aware Policy Learning in Contextual Banditscited by 0