# MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search

arXiv: https://arxiv.org/abs/2412.20086 · HTML: https://mlsanity.com/paper/2412.20086/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 28 Dec 2024.

## Builds on (most cited first, 25 of 25)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Simple Black-box Adversarial Attacks](https://mlsanity.com/paper/1905.07121/) (2019)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [Fairness in Deep Learning: A Computational Perspective](https://mlsanity.com/paper/1908.08843/) (2019)
- [Satisfying Real-world Goals with Dataset Constraints](https://mlsanity.com/paper/1606.07558/) (2016)
- [The Price of Fair PCA: One Extra Dimension](https://mlsanity.com/paper/1811.00103/) (2018)
- [Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques](https://mlsanity.com/paper/1901.03116/) (2019)
- [FairRec: Fairness-aware News Recommendation with Decomposed Adversarial Learning](https://mlsanity.com/paper/2006.16742/) (2020)
- [NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification](https://mlsanity.com/paper/2112.13214/) (2021)
- [Penalizing Unfairness in Binary Classification](https://mlsanity.com/paper/1707.00044/) (2017)
- [Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testing](https://mlsanity.com/paper/2305.11602/) (2023)
- [Automated Test Generation to Detect Individual Discrimination in AI Models](https://mlsanity.com/paper/1809.03260/) (2018)
- [VerifyMed -- A blockchain platform for transparent trust in virtualized healthcare: Proof-of-concept](https://mlsanity.com/paper/2005.08804/) (2020)
- [On Fair Selection in the Presence of Implicit Variance](https://mlsanity.com/paper/2006.13699/) (2020)
- [Astraea: Grammar-based Fairness Testing](https://mlsanity.com/paper/2010.02542/) (2020)
- [On Fair Selection in the Presence of Implicit and Differential Variance](https://mlsanity.com/paper/2112.05630/) (2021)
- [Fairness Testing of Deep Image Classification with Adequacy Metrics](https://mlsanity.com/paper/2111.08856/) (2021)
- [Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances](https://mlsanity.com/paper/2209.08321/) (2022)

## Built on by (most cited first, 4 of 4)

- [Testing Individual Fairness in Graph Neural Networks](https://mlsanity.com/paper/2504.18353/) (2025)
- [Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations](https://mlsanity.com/paper/2512.00556/) (2025)
- [Provable Fairness Repair for Deep Neural Networks](https://mlsanity.com/paper/2605.19549/) (2026)
- [Fairness Invariants: A Relational Approach to Explaining and Mitigating Fairness Bugs](https://mlsanity.com/paper/2608.26209/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have shown powerful performance in various applications and are increasingly being used in decision-making systems. However, concerns about fairness in DNNs always persist. Some efficient white-box fairness testing methods about individual fairness have been proposed. Nevertheless, the development of black-box methods has stagnated, and the performance of existing methods is far behind that of white-box methods. In this paper, we propose a novel black-box individual fairness testing method called Model-Agnostic Fairness Testing (MAFT). By leveraging MAFT, practitioners can effectively identify and address discrimination in DL models, regardless of the specific algorithm or architecture employed. Our approach adopts lightweight procedures such as gradient estimation and attribute perturbation rather than non-trivial procedures like symbol execution, rendering it significantly more scalable and applicable than existing methods. We demonstrate that MAFT achieves the same effectiveness as state-of-the-art white-box methods whilst improving the applicability to large-scale networks. Compared to existing black-box approaches, our approach demonstrates distinguished performance in discovering fairness violations w.r.t effectiveness (approximately 14.69 times) and efficiency (approximately 32.58 times).