mlsanityData to 14 Oct 2026

Auditing Fairness by Betting

arXiv 2305.17570 ↗First posted 27 May 2023stat.ML cs.AI cs.CYcited by 25

Not on the map. Why?

Abstract (arXiv)

We provide practical, efficient, and nonparametric methods for auditing the fairness of deployed classification and regression models. Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems. We also allow the data to be collected by a probabilistic policy as opposed to sampled uniformly from the population. This enables auditing to be conducted on data gathered for another purpose. Moreover, this policy may change over time and different policies may be used on different subpopulations. Finally, our methods can handle distribution shift resulting from either changes to the model or changes in the underlying population. Our approach is based on recent progress in anytime-valid inference and game-theoretic statistics-the "testing by betting" framework in particular. These connections ensure that our methods are interpretable, fast, and easy to implement. We demonstrate the efficacy of our approach on three benchmark fairness datasets.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 29 of its 63 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

29 of 63 references.

References · 29

  1. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  2. 2018Recognition in Terra Incognitacited by 658
  3. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 449
  4. 2017Algorithmic decision making and the cost of fairnesscited by 403
  5. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  6. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  7. 2020Estimating means of bounded random variables by bettingcited by 169
  8. 2022Game-theoretic statistics and safe anytime-valid inferencecited by 155
  9. 2020Algorithmic Fairnesscited by 143
  10. 2016Coin Betting and Parameter-Free Online Learningcited by 133
  11. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 125
  12. 2019Safe Testingcited by 101
17 more
  1. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  2. 2021There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"cited by 53
  3. 2020Admissible anytime-valid sequential inference must rely on nonnegative martingalescited by 48
  4. 2021Nonparametric Two-Sample Testing by Bettingcited by 37
  5. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  6. 2019Parameter-Free Online Convex Optimization with Sub-Exponential Noisecited by 28
  7. 2020A Statistical Test for Probabilistic Fairnesscited by 25
  8. 2021Testing Group Fairness via Optimal Transport Projectionscited by 23
  9. 2021Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studiescited by 22
  10. 2020Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Bettingcited by 19
  11. 2022A composite generalization of Ville's martingale theorem using e-processescited by 10
  12. 2021Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policycited by 7
  13. 2021RiLACS: Risk-Limiting Audits via Confidence Sequencescited by 6
  14. 2023Risk-limiting Financial Audits via Weighted Sampling without Replacementcited by 6
  15. 2020Interactive rank testing by bettingcited by 5
  16. 2022Integrating Reward Maximization and Population Estimation: Sequential Decision-Making for Internal Revenue Service Audit Selectioncited by 5
  17. 2022Entropy Regularization for Population Estimationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 14.

25 papers.

Sort:

By area:

  1. 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 17
  2. 2024PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attackscited by 11
  3. 2024Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Bettingcited by 11
  4. 2025On Stopping Times of Power-one Sequential Tests: Tight Lower and Upper Boundscited by 7
  5. 2025Robust ML Auditing using Prior Knowledgecited by 6
  6. 2024Under manipulations, are some AI models harder to audit?cited by 5
  7. 2025Sequentially Auditing Differential Privacycited by 5
  8. 2025Optimistic Interior Point Methods for Sequential Hypothesis Testing by Bettingcited by 4
  9. 2025Stopping Rules for Stochastic Gradient Descent via Anytime-Valid Confidence Sequencescited by 4
  10. 2026Testing For Distribution Shifts with Conditional Conformal Test Martingalescited by 4
  11. 2025From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harmscited by 3
  12. 2026Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Boundscited by 2
13 more
  1. 2026Auditing Fairness under Model Updates: Fundamental Complexity and Property-Preserving Updatescited by 1
  2. 2026Global Sequential Testing for Multi-Stream Auditingcited by 1
  3. 2026Multi-Armed Sequential Hypothesis Testing by Bettingcited by 1
  4. 2026The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regimecited by 1
  5. 2026Optimal Rates for Differentially Private Hypothesis Testing with E-valuescited by 1
  6. 2025Cost Efficient Fairness Audit Under Partial Feedbackcited by 0
  7. 2026GAAVI: Global Asymptotic Anytime Valid Inference for the Conditional Mean Functioncited by 0
  8. 2026Incentive Aware AI Regulations: A Credal Characterisationcited by 0
  9. 2026Sequential Audit Sampling for Finite Populations with Exact and Simulation-based Guaranteecited by 0
  10. 2026The Evaluation Game: Beyond Static LLM Benchmarkingcited by 0
  11. 2026Fairness Auditing: Lower Bounds on Company Manipulationcited by 0
  12. 2026Always-On Experimentationcited by 0
  13. 2026Bayesian Learning to Select Fair Classifierscited by 0