mlsanityData to 7 Oct 2026

Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment

arXiv 2109.11679 ↗First posted 22 Sep 2021stat.ML cs.LG stat.MEcited by 10

Not on the map. Why?

Abstract (arXiv)

Algorithmic recommendations and decisions have become ubiquitous in today's society. Many of these data-driven policies, especially in the realm of public policy, are based on known, deterministic rules to ensure their transparency and interpretability. We examine a particular case of algorithmic pre-trial risk assessments in the US criminal justice system, which provide deterministic classification scores and recommendations to help judges make release decisions. Our goal is to analyze data from a unique field experiment on an algorithmic pre-trial risk assessment to investigate whether the scores and recommendations can be improved. Unfortunately, prior methods for policy learning are not applicable because they require existing policies to be stochastic. We develop a maximin robust optimization approach that partially identifies the expected utility of a policy, and then finds a policy that maximizes the worst-case expected utility. The resulting policy has a statistical safety property, limiting the probability of producing a worse policy than the existing one, under structural assumptions about the outcomes. Our analysis of data from the field experiment shows that we can safely improve certain components of the risk assessment instrument by classifying arrestees as lower risk under a wide range of utility specifications, though the analysis is not informative about several components of the instrument.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 19 of its 61 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

19 of 61 references.

References · 19

  1. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
  2. 2017On Kernelized Multi-armed Banditscited by 348
  3. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  4. 2017Quasi-Oracle Estimation of Heterogeneous Treatment Effectscited by 328
  5. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
  6. 2017Mostly Exploration-Free Algorithms for Contextual Banditscited by 85
  7. 2013Counterfactual Probabilities: Computational Methods, Bounds and Applicationscited by 67
  8. 2019Counterfactual Risk Assessments, Evaluation, and Fairnesscited by 59
  9. 2018The age of secrecy and unfairness in recidivism predictioncited by 53
  10. 2017Balanced Policy Evaluation and Learningcited by 51
  11. 2021Practical and Rigorous Uncertainty Bounds for Gaussian Process Regressioncited by 49
  12. 2017Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effectscited by 41
7 more
  1. 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
  2. 2019A semiparametric instrumental variable approach to optimal treatment regimes under endogeneitycited by 22
  3. 2020Principal Fairness for Human and Algorithmic Decision-Makingcited by 20
  4. 2020Greedy Algorithm almost Dominates in Smoothed Contextual Banditscited by 20
  5. 2020Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessmentcited by 18
  6. 2020Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approachcited by 17
  7. 2017Optimized Regression Discontinuity Designscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 4; 2024: 4.

10 papers.

  1. 2022Policy Learning with Asymmetric Counterfactual Utilitiescited by 9
  2. 2023Individualized Policy Evaluation and Learning under Clustered Network Interferencecited by 6
  3. 2023Policy Learning with Distributional Welfarecited by 5
  4. 2024Minimax Regret Estimation for Generalizing Heterogeneous Treatment Effects with Multisite Datacited by 5
  5. 2023Bayesian Safe Policy Learning with Chance Constrained Optimization: Application to Military Security Assessment during the Vietnam Warcited by 4
  6. 2024Designing Algorithmic Recommendations to Achieve Human-AI Complementaritycited by 4
  7. 2022Distributionally Robust Causal Inference with Observational Datacited by 2
  8. 2024CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policiescited by 2
  9. 2023Mind the Gap: Optimal and Equitable Encouragement Policiescited by 0
  10. 2024Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rulescited by 0