# FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods

arXiv: https://arxiv.org/abs/2306.09468 · HTML: https://mlsanity.com/paper/2306.09468/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Censoring Representations with an Adversary](https://mlsanity.com/paper/1511.05897/) (2015)
- [Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations](https://mlsanity.com/paper/1707.00075/) (2017)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Wasserstein Fair Classification](https://mlsanity.com/paper/1907.12059/) (2019)
- [Fair Mixup: Fairness via Interpolation](https://mlsanity.com/paper/2103.06503/) (2021)
- [Learning to Pivot with Adversarial Networks](https://mlsanity.com/paper/1611.01046/) (2016)
- [Fair and Optimal Classification via Post-Processing](https://mlsanity.com/paper/2211.01528/) (2022)
- [Average Individual Fairness: Algorithms, Generalization and Experiments](https://mlsanity.com/paper/1905.10607/) (2019)
- [Achieving Fairness at No Utility Cost via Data Reweighing with Influence](https://mlsanity.com/paper/2202.00787/) (2022)
- [R'enyi Fair Inference](https://mlsanity.com/paper/1906.12005/) (2019)

## Built on by (most cited first, 30 of 33)

- [AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark](https://mlsanity.com/paper/2406.00783/) (2024)
- [Differentially Private Tabular Data Synthesis using Large Language Models](https://mlsanity.com/paper/2406.01457/) (2024)
- [Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts](https://mlsanity.com/paper/2406.17974/) (2024)
- [OxonFair: A Flexible Toolkit for Algorithmic Fairness](https://mlsanity.com/paper/2407.13710/) (2024)
- [CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Models](https://mlsanity.com/paper/2408.09819/) (2024)
- [fairret: a Framework for Differentiable Fairness Regularization Terms](https://mlsanity.com/paper/2310.17256/) (2023)
- [ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods](https://mlsanity.com/paper/2409.16965/) (2024)
- [When Fairness Meets Privacy: Exploring Privacy Threats in Fair Binary Classifiers via Membership Inference Attacks](https://mlsanity.com/paper/2311.03865/) (2023)
- [Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework](https://mlsanity.com/paper/2508.15193/) (2025)
- [On the Maximal Local Disparity of Fairness-Aware Classifiers](https://mlsanity.com/paper/2406.03255/) (2024)
- [Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML](https://mlsanity.com/paper/2411.11101/) (2024)
- [Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective](https://mlsanity.com/paper/2506.07861/) (2025)
- [Towards Practical Non-Adversarial Distribution Matching](https://mlsanity.com/paper/2310.19690/) (2023)
- [Learning Fairer Representations with FairVIC](https://mlsanity.com/paper/2404.18134/) (2024)
- [Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challenges](https://mlsanity.com/paper/2409.04373/) (2024)
- [Combining Machine Learning Defenses without Conflicts](https://mlsanity.com/paper/2411.09776/) (2024)
- [Fairness via Independence: A (Conditional) Distance Covariance Framework](https://mlsanity.com/paper/2412.00720/) (2024)
- [Do Fairness Interventions Come at the Cost of Privacy: Evaluations for Binary Classifiers](https://mlsanity.com/paper/2503.06150/) (2025)
- [TABFAIRGDT: A Fast Fair Tabular Data Generator using Autoregressive Decision Trees](https://mlsanity.com/paper/2509.19927/) (2025)
- [Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphs](https://mlsanity.com/paper/2510.18473/) (2025)
- [Bias Begins with Data: The FairGround Corpus for Robust and Reproducible Research on Algorithmic Fairness](https://mlsanity.com/paper/2510.22363/) (2025)
- [debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias](https://mlsanity.com/paper/2410.13146/) (2024)
- [LoGoFair: Post-Processing for Local and Global Fairness in Federated Learning](https://mlsanity.com/paper/2503.17231/) (2025)
- [Fair-FLIP: Fair Deepfake Detection with Fairness-Oriented Final Layer Input Prioritising](https://mlsanity.com/paper/2507.08912/) (2025)
- [Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs](https://mlsanity.com/paper/2602.03895/) (2026)
- [On Demographic Group Fairness Guarantees in Deep Learning](https://mlsanity.com/paper/2412.20377/) (2024)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [FedFACT: A Provable Framework for Controllable Group-Fairness Calibration in Federated Learning](https://mlsanity.com/paper/2506.03777/) (2025)
- [Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training of Deep Neural Networks](https://mlsanity.com/paper/2507.04033/) (2025)
- [PAC-Bayesian Generalization Guarantees for Fairness on Stochastic and Deterministic Classifiers](https://mlsanity.com/paper/2602.11722/) (2026)

## Abstract (arXiv)

This paper introduces the Fair Fairness Benchmark (textsfFFB), a benchmarking framework for in-processing group fairness methods. Ensuring fairness in machine learning is important for ethical compliance. However, there exist challenges in comparing and developing fairness methods due to inconsistencies in experimental settings, lack of accessible algorithmic implementations, and limited extensibility of current fairness packages and tools. To address these issues, we introduce an open-source standardized benchmark for evaluating in-processing group fairness methods and provide a comprehensive analysis of state-of-the-art methods to ensure different notions of group fairness. This work offers the following key contributions: the provision of flexible, extensible, minimalistic, and research-oriented open-source code; the establishment of unified fairness method benchmarking pipelines; and extensive benchmarking, which yields key insights from 45,079 experiments, 14,428 GPU hours. We believe that our work will significantly facilitate the growth and development of the fairness research community.