# Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment

arXiv: https://arxiv.org/abs/2109.11679 · HTML: https://mlsanity.com/paper/2109.11679/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 61 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [On Kernelized Multi-armed Bandits](https://mlsanity.com/paper/1704.00445/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Mostly Exploration-Free Algorithms for Contextual Bandits](https://mlsanity.com/paper/1704.09011/) (2017)
- [Counterfactual Probabilities: Computational Methods, Bounds and Applications](https://mlsanity.com/paper/1302.6784/) (2013)
- [Counterfactual Risk Assessments, Evaluation, and Fairness](https://mlsanity.com/paper/1909.00066/) (2019)
- [The age of secrecy and unfairness in recidivism prediction](https://mlsanity.com/paper/1811.00731/) (2018)
- [Balanced Policy Evaluation and Learning](https://mlsanity.com/paper/1705.07384/) (2017)
- [Practical and Rigorous Uncertainty Bounds for Gaussian Process Regression](https://mlsanity.com/paper/2105.02796/) (2021)
- [Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects](https://mlsanity.com/paper/1706.09523/) (2017)
- [Learning Optimal Distributionally Robust Individualized Treatment Rules](https://mlsanity.com/paper/2006.15121/) (2020)
- [A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity](https://mlsanity.com/paper/1911.09260/) (2019)
- [Principal Fairness for Human and Algorithmic Decision-Making](https://mlsanity.com/paper/2005.10400/) (2020)
- [Greedy Algorithm almost Dominates in Smoothed Contextual Bandits](https://mlsanity.com/paper/2005.10624/) (2020)
- [Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment](https://mlsanity.com/paper/2012.02845/) (2020)
- [Estimating Optimal Treatment Rules with an Instrumental Variable: A Partial Identification Learning Approach](https://mlsanity.com/paper/2002.02579/) (2020)
- [Optimized Regression Discontinuity Designs](https://mlsanity.com/paper/1705.01677/) (2017)

## Built on by (most cited first, 10 of 10)

- [Policy Learning with Asymmetric Counterfactual Utilities](https://mlsanity.com/paper/2206.10479/) (2022)
- [Individualized Policy Evaluation and Learning under Clustered Network Interference](https://mlsanity.com/paper/2311.02467/) (2023)
- [Policy Learning with Distributional Welfare](https://mlsanity.com/paper/2311.15878/) (2023)
- [Minimax Regret Estimation for Generalizing Heterogeneous Treatment Effects with Multisite Data](https://mlsanity.com/paper/2412.11136/) (2024)
- [Bayesian Safe Policy Learning with Chance Constrained Optimization: Application to Military Security Assessment during the Vietnam War](https://mlsanity.com/paper/2307.08840/) (2023)
- [Designing Algorithmic Recommendations to Achieve Human-AI Complementarity](https://mlsanity.com/paper/2405.01484/) (2024)
- [Distributionally Robust Causal Inference with Observational Data](https://mlsanity.com/paper/2210.08326/) (2022)
- [CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policies](https://mlsanity.com/paper/2408.12004/) (2024)
- [Mind the Gap: Optimal and Equitable Encouragement Policies](https://mlsanity.com/paper/2309.07176/) (2023)
- [Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rules](https://mlsanity.com/paper/2404.17019/) (2024)

## Abstract (arXiv)

Algorithmic recommendations and decisions have become ubiquitous in today's society. Many of these data-driven policies, especially in the realm of public policy, are based on known, deterministic rules to ensure their transparency and interpretability. We examine a particular case of algorithmic pre-trial risk assessments in the US criminal justice system, which provide deterministic classification scores and recommendations to help judges make release decisions. Our goal is to analyze data from a unique field experiment on an algorithmic pre-trial risk assessment to investigate whether the scores and recommendations can be improved. Unfortunately, prior methods for policy learning are not applicable because they require existing policies to be stochastic. We develop a maximin robust optimization approach that partially identifies the expected utility of a policy, and then finds a policy that maximizes the worst-case expected utility. The resulting policy has a statistical safety property, limiting the probability of producing a worse policy than the existing one, under structural assumptions about the outcomes. Our analysis of data from the field experiment shows that we can safely improve certain components of the risk assessment instrument by classifying arrestees as lower risk under a wide range of utility specifications, though the analysis is not informative about several components of the instrument.