# Active Fairness Auditing

arXiv: https://arxiv.org/abs/2206.08450 · HTML: https://mlsanity.com/paper/2206.08450/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 40 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 8 of 8)

- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Beyond Disagreement-based Agnostic Active Learning](https://mlsanity.com/paper/1407.2657/) (2014)
- [Efficient and Parsimonious Agnostic Active Learning](https://mlsanity.com/paper/1506.08669/) (2015)
- [Estimating Learnability in the Sublinear Data Regime](https://mlsanity.com/paper/1805.01626/) (2018)
- [VC Dimension and Distribution-Free Sample-Based Testing](https://mlsanity.com/paper/2012.03923/) (2020)
- [Active Tolerant Testing](https://mlsanity.com/paper/1711.00388/) (2017)
- [Auditing: Active Learning with Outcome-Dependent Query Costs](https://mlsanity.com/paper/1306.2347/) (2013)
- [Estimating decision tree learnability with polylogarithmic sample complexity](https://mlsanity.com/paper/2011.01584/) (2020)

## Built on by (most cited first, 27 of 27)

- [Statistical Inference for Fairness Auditing](https://mlsanity.com/paper/2305.03712/) (2023)
- [Demystifying Local and Global Fairness Trade-offs in Federated Learning Using Partial Information Decomposition](https://mlsanity.com/paper/2307.11333/) (2023)
- [Log Probability Tracking of LLM APIs](https://mlsanity.com/paper/2512.03816/) (2025)
- [Can Querying for Bias Leak Protected Attributes? Achieving Privacy With Smooth Sensitivity](https://mlsanity.com/paper/2211.02139/) (2022)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [A Generic Framework for Conformal Fairness](https://mlsanity.com/paper/2505.16115/) (2025)
- [Active Fourier Auditor for Estimating Distributional Properties of ML Models](https://mlsanity.com/paper/2410.08111/) (2024)
- [Robust ML Auditing using Prior Knowledge](https://mlsanity.com/paper/2505.04796/) (2025)
- [Under manipulations, are some AI models harder to audit?](https://mlsanity.com/paper/2402.09043/) (2024)
- [A Brief Tutorial on Sample Size Calculations for Fairness Audits](https://mlsanity.com/paper/2312.04745/) (2023)
- [Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks](https://mlsanity.com/paper/2507.20708/) (2025)
- [A Unified View of Group Fairness Tradeoffs Using Partial Information Decomposition](https://mlsanity.com/paper/2406.04562/) (2024)
- [A Comprehensive Survey and Classification of Evaluation Criteria for Trustworthy Artificial Intelligence](https://mlsanity.com/paper/2410.17281/) (2024)
- [Adaptive auditing of AI systems with anytime-valid guarantees](https://mlsanity.com/paper/2605.07002/) (2026)
- [Cross-GAN Auditing: Unsupervised Identification of Attribute Level Similarities and Differences between Pretrained Generative Models](https://mlsanity.com/paper/2303.10774/) (2023)
- [Fairness Auditing with Multi-Agent Collaboration](https://mlsanity.com/paper/2402.08522/) (2024)
- [ExpProof : Operationalizing Explanations for Confidential Models with ZKPs](https://mlsanity.com/paper/2502.03773/) (2025)
- [Conformal Prediction: A Theoretical Note and Benchmarking Transductive Node Classification in Graphs](https://mlsanity.com/paper/2409.18332/) (2024)
- [Auditing Fairness under Model Updates: Fundamental Complexity and Property-Preserving Updates](https://mlsanity.com/paper/2601.05909/) (2026)
- [Peer-induced Fairness: A Causal Approach for Algorithmic Fairness Auditing](https://mlsanity.com/paper/2408.02558/) (2024)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [Sublinear Algorithms for Wasserstein and Total Variation Distances: Applications to Fairness and Privacy Auditing](https://mlsanity.com/paper/2503.07775/) (2025)
- [The Fair Game: Auditing & Debiasing AI Algorithms Over Time](https://mlsanity.com/paper/2508.06443/) (2025)
- [Cost Efficient Fairness Audit Under Partial Feedback](https://mlsanity.com/paper/2510.03734/) (2025)
- [The Evaluation Game: Beyond Static LLM Benchmarking](https://mlsanity.com/paper/2605.19377/) (2026)
- [Fairness Auditing: Lower Bounds on Company Manipulation](https://mlsanity.com/paper/2608.00568/) (2026)
- [Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening](https://mlsanity.com/paper/2610.01005/) (2026)

## Abstract (arXiv)

The fast spreading adoption of machine learning (ML) by companies across industries poses significant regulatory challenges. One such challenge is scalability: how can regulatory bodies efficiently audit these ML models, ensuring that they are fair? In this paper, we initiate the study of query-based auditing algorithms that can estimate the demographic parity of ML models in a query-efficient manner. We propose an optimal deterministic algorithm, as well as a practical randomized, oracle-efficient algorithm with comparable guarantees. Furthermore, we make inroads into understanding the optimal query complexity of randomized active fairness estimation algorithms. Our first exploration of active fairness estimation aims to put AI governance on firmer theoretical foundations.