mlsanityData to 7 Oct 2026

Minimax Optimal Estimation of Stability Under Distribution Shift

arXiv 2212.06338 ↗First posted 13 Dec 2022stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The performance of decision policies and prediction models often deteriorates when applied to environments different from the ones seen during training. To ensure reliable operation, we analyze the stability of a system under distribution shift, which is defined as the smallest change in the underlying environment that causes the system's performance to deteriorate beyond a permissible threshold. In contrast to standard tail risk measures and distributionally robust losses that require the specification of a plausible magnitude of distribution shift, the stability measure is defined in terms of a more intuitive quantity: the level of acceptable performance degradation. We develop a minimax optimal estimator of stability and analyze its convergence rate, which exhibits a fundamental phase shift behavior. Our characterization of the minimax convergence rate shows that evaluating stability against large performance degradation incurs a statistical cost. Empirically, we demonstrate the practical utility of our stability framework by using it to compare system designs on problems where robustness to distribution shift is critical.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 154 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

34 of 154 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016OpenAI Gymcited by 1,684
  6. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  7. 2017Deep Reinforcement Learning that Matterscited by 727
  8. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  9. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  10. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  12. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
22 more
  1. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
  2. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  3. 2016Distributionally Robust Stochastic Optimization with Wasserstein Distancecited by 182
  4. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
  5. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  6. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  7. 2013Stabilitycited by 119
  8. 2020Ethical Machine Learning in Health Carecited by 112
  9. 2020The iWildCam 2020 Competition Datasetcited by 86
  10. 2018Confounding-Robust Policy Improvementcited by 73
  11. 2020Distributionally Robust Losses for Latent Covariate Mixturescited by 57
  12. 2020Queueing Network Controls via Deep Reinforcement Learningcited by 40
  13. 2018Bounds on the conditional and average treatment effect with unobserved confounding factorscited by 39
  14. 2017From Data to Decisions: Distributionally Robust Optimization is Optimalcited by 36
  15. 2016The Empirical Likelihood Approach to Quantifying Uncertainty in Sample Average Approximationcited by 12
  16. 2021The s-value: evaluating stability with respect to distributional shiftscited by 10
  17. 2021Learning and Information in Stochastic Networks and Queuescited by 8
  18. 2016Sensitivity to Serial Dependency of Input Processes: A Robust Approachcited by 6
  19. 2020Assessing External Validity Over Worst-case Subpopulationscited by 5
  20. 2019Driver Surge Pricingcited by 4
  21. 2017Optimization-based Quantification of Simulation Input Uncertainty via Empirical Likelihoodcited by 2
  22. 2019From Local to Global: External Validity in a Fertility Natural Experimentcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  2. 2023Confounding-Robust Policy Improvement with Human-AI Teamscited by 4
  3. 2024Distributionally robust risk evaluation with an isotonic constraintcited by 3
  4. 2024Stability Evaluation via Distributional Perturbation Analysiscited by 1