# Adversarial Scrutiny of Evidentiary Statistical Software

arXiv: https://arxiv.org/abs/2206.09305 · HTML: https://mlsanity.com/paper/2206.09305/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 115 references that are in the map.
- First posted on arXiv 19 Jun 2022.

## Builds on (most cited first, 15 of 15)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Outcome Indistinguishability](https://mlsanity.com/paper/2011.13426/) (2020)
- [Roles for Computing in Social Change](https://mlsanity.com/paper/1912.04883/) (2019)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Hyperparameter Optimization Is Deceiving Us, and How to Stop It](https://mlsanity.com/paper/2102.03034/) (2021)
- [A Silicon Valley Love Triangle: Hiring Algorithms, Pseudo-Science, and the Quest for Auditability](https://mlsanity.com/paper/2106.12403/) (2021)
- [Patterns, predictions, and actions: A story about machine learning](https://mlsanity.com/paper/2102.05242/) (2021)
- [An Uncommon Task: Participatory Design in Legal AI](https://mlsanity.com/paper/2203.06246/) (2022)
- [Defending IEEE Software Standards in Federal Criminal Court](https://mlsanity.com/paper/2106.11021/) (2021)

## Built on by (most cited first, 5 of 5)

- [Algorithmic Collective Action in Machine Learning](https://mlsanity.com/paper/2302.04262/) (2023)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court](https://mlsanity.com/paper/2403.13004/) (2024)
- [Greedy PIG: Adaptive Integrated Gradients](https://mlsanity.com/paper/2311.06192/) (2023)
- [Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software](https://mlsanity.com/paper/2604.10875/) (2026)

## Abstract (arXiv)

The U.S. criminal legal system increasingly relies on software output to convict and incarcerate people. In a large number of cases each year, the government makes these consequential decisions based on evidence from statistical software -- such as probabilistic genotyping, environmental audio detection, and toolmark analysis tools -- that defense counsel cannot fully cross-examine or scrutinize. This undermines the commitments of the adversarial criminal legal system, which relies on the defense's ability to probe and test the prosecution's case to safeguard individual rights. Responding to this need to adversarially scrutinize output from such software, we propose robust adversarial testing as an audit framework to examine the validity of evidentiary statistical software. We define and operationalize this notion of robust adversarial testing for defense use by drawing on a large body of recent work in robust machine learning and algorithmic fairness. We demonstrate how this framework both standardizes the process for scrutinizing such tools and empowers defense lawyers to examine their validity for instances most relevant to the case at hand. We further discuss existing structural and institutional challenges within the U.S. criminal legal system that may create barriers for implementing this and other such audit frameworks and close with a discussion on policy changes that could help address these concerns.