# Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software

arXiv: https://arxiv.org/abs/2604.10875 · HTML: https://mlsanity.com/paper/2604.10875/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 77 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 20 of 20)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance](https://mlsanity.com/paper/2206.04737/) (2022)
- [Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem](https://mlsanity.com/paper/2310.02521/) (2023)
- [AI auditing: The Broken Bus on the Road to AI Accountability](https://mlsanity.com/paper/2401.14462/) (2024)
- [Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices](https://mlsanity.com/paper/2006.12358/) (2020)
- [Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits](https://mlsanity.com/paper/2205.06922/) (2022)
- [Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling](https://mlsanity.com/paper/2402.17861/) (2024)
- [Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics](https://mlsanity.com/paper/2202.08792/) (2022)
- [This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology](https://mlsanity.com/paper/1909.11869/) (2019)
- [Auditing Work: Exploring the New York City algorithmic bias audit regime](https://mlsanity.com/paper/2402.08101/) (2024)
- [Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising](https://mlsanity.com/paper/2308.15768/) (2023)
- [Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability](https://mlsanity.com/paper/2406.01399/) (2024)
- [A Framework for Assurance Audits of Algorithmic Systems](https://mlsanity.com/paper/2401.14908/) (2024)
- [A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations](https://mlsanity.com/paper/2401.17486/) (2024)
- [Adversarial Scrutiny of Evidentiary Statistical Software](https://mlsanity.com/paper/2206.09305/) (2022)
- [Defending IEEE Software Standards in Federal Criminal Court](https://mlsanity.com/paper/2106.11021/) (2021)
- [(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court](https://mlsanity.com/paper/2403.13004/) (2024)

## Abstract (arXiv)

AI governance efforts increasingly rely on audit standards: agreed-upon practices for conducting audits. However, poorly designed standards can hide and lend credibility to inadequate systems. We explore how an audit standard's design influences its effectiveness through a case study of ASB 018, a standard for auditing probabilistic genotyping software -- software that the U.S. criminal legal system increasingly uses to analyze DNA samples. Through qualitative analysis of ASB 018 and five audit reports, we identify numerous gaps between the standard's desired outcomes and the auditing practices it enables. For instance, ASB 018 envisions that compliant audits establish restrictions on software use based on observed failures. However, audits can comply without establishing such boundaries. We connect these gaps to the design of the standard's requirements such as vague language and undefined terms. We conclude with recommendations for designing audit standards and evaluating their effectiveness.