# Fairness Audits of Institutional Risk Models in Deployed ML Pipelines

arXiv: https://arxiv.org/abs/2604.19468 · HTML: https://mlsanity.com/paper/2604.19468/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 8 of 8)

- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Difficult Lessons on Social Prediction from Wisconsin Public Schools](https://mlsanity.com/paper/2304.06205/) (2023)
- [A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare](https://mlsanity.com/paper/2107.03487/) (2021)
- [A Human-Centered Review of Algorithms in Homelessness Research](https://mlsanity.com/paper/2401.13247/) (2024)
- ["This is not a data problem": Algorithms and Power in Public Higher Education in Canada](https://mlsanity.com/paper/2403.13969/) (2024)
- [A Human-Centered Review of Algorithms in Decision-Making in Higher Education](https://mlsanity.com/paper/2302.05839/) (2023)

## Abstract (arXiv)

Fairness audits of institutional risk models are critical for understanding how deployed machine learning pipelines allocate resources. Drawing on multi-year collaboration with Centennial College, where our prior ethnographic work introduced the ASP-HEI Cycle, we present a replica-based audit of a deployed Early Warning System (EWS), replicating its model using institutional training data and design specifications. We evaluate disparities by gender, age, and residency status across the full pipeline (training data, model predictions, and post-processing) using standard fairness metrics. Our audit reveals systematic misallocation: younger, male, and international students are disproportionately flagged for support, even when many ultimately succeed, while older and female students with comparable dropout risk are under-identified. Post-processing amplifies these disparities by collapsing heterogeneous probabilities into percentile-based risk tiers. This work provides a replicable methodology for auditing institutional ML systems and shows how disparities emerge and compound across stages, highlighting the importance of evaluating construct validity alongside statistical fairness. It contributes one empirical thread to a broader program investigating algorithms, student data, and power in higher education.