Sequential Fairness Auditing with Limited Output Access
Not on the map. Why?
Abstract (arXiv)
External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited access to deployed models and must rely on query-based interactions. Most existing fairness evaluation methods assume static datasets and fixed-sample statistical tests, making them poorly suited to real-world auditing scenarios in which evidence must be collected sequentially under query constraints. In this work, we formulate fairness auditing as a tolerance-aware sequential hypothesis-testing problem under limited model output access. We develop a sequential generalized likelihood-ratio framework that allows auditors to accumulate evidence from a finite audit pool and stop once sufficient support for compliance or violation has been obtained. The framework is instantiated for decision-based Statistical Parity and Equal Opportunity audits, and extended to score- and logit-based proxy audits when richer observables are available. Our results show that both the fairness metric and the level of model access significantly affect audit efficiency, and that the benefits of richer output information are not uniform across auditing settings. In particular, richer outputs can substantially reduce the number of queries required for some fairness metrics and operating regimes, while offering limited gains in near-threshold cases. This work provides a practical statistical framework for sequential fairness auditing under realistic deployment constraints.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
12 of 17 references.
References · 12
- 2014Adam: A Method for Stochastic Optimizationcited by 37,093
- 2016Equality of Opportunity in Supervised Learningcited by 2,005
- 2014Certifying and removing disparate impactcited by 715
- 2018A Reductions Approach to Fair Classificationcited by 585
- 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 444
- 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 381
- 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 224
- 2022Game-theoretic statistics and safe anytime-valid inferencecited by 149
- 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 125
- 2020A Statistical Test for Probabilistic Fairnesscited by 25
- 2024From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditingcited by 15
- 2026Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMscited by 6
Built on by
Not cited yet.