# Fairness Auditing with Multi-Agent Collaboration

arXiv: https://arxiv.org/abs/2402.08522 · HTML: https://mlsanity.com/paper/2402.08522/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [A Clarification of the Nuances in the Fairness Metrics Landscape](https://mlsanity.com/paper/2106.00467/) (2021)
- [Federated Mutual Learning](https://mlsanity.com/paper/2006.16765/) (2020)
- [A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges](https://mlsanity.com/paper/2305.06969/) (2023)
- [From Fair Decision Making to Social Equality](https://mlsanity.com/paper/1812.02952/) (2018)
- [Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mutual Information](https://mlsanity.com/paper/2104.09460/) (2021)
- [Active Fairness Auditing](https://mlsanity.com/paper/2206.08450/) (2022)
- [Inherent Limitations of AI Fairness](https://mlsanity.com/paper/2212.06495/) (2022)
- [Algorithmic audits of algorithms, and the law](https://mlsanity.com/paper/2203.03711/) (2022)
- [Crowd-assisted Search for Price Discrimination in E-Commerce: First results](https://mlsanity.com/paper/1307.4531/) (2013)
- [Beyond Demographic Parity: Redefining Equal Treatment](https://mlsanity.com/paper/2303.08040/) (2023)

## Built on by (most cited first, 2 of 2)

- [Robust ML Auditing using Prior Knowledge](https://mlsanity.com/paper/2505.04796/) (2025)
- [P2NIA: Privacy-Preserving Non-Iterative Auditing](https://mlsanity.com/paper/2504.00874/) (2025)

## Abstract (arXiv)

Existing work in fairness auditing assumes that each audit is performed independently. In this paper, we consider multiple agents working together, each auditing the same platform for different tasks. Agents have two levers: their collaboration strategy, with or without coordination beforehand, and their strategy for sampling appropriate data points. We theoretically compare the interplay of these levers. Our main findings are that (i) collaboration is generally beneficial for accurate audits, (ii) basic sampling methods often prove to be effective, and (iii) counter-intuitively, extensive coordination on queries often deteriorates audits accuracy as the number of agents increases. Experiments on three large datasets confirm our theoretical results. Our findings motivate collaboration during fairness audits of platforms that use ML models for decision-making.