mlsanityData to 7 Oct 2026

FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods

arXiv 2306.09468 ↗First posted 15 Jun 2023cs.LG cs.AI cs.CYcited by 33

Not on the map. Why?

Abstract (arXiv)

This paper introduces the Fair Fairness Benchmark (textsfFFB), a benchmarking framework for in-processing group fairness methods. Ensuring fairness in machine learning is important for ethical compliance. However, there exist challenges in comparing and developing fairness methods due to inconsistencies in experimental settings, lack of accessible algorithmic implementations, and limited extensibility of current fairness packages and tools. To address these issues, we introduce an open-source standardized benchmark for evaluating in-processing group fairness methods and provide a comprehensive analysis of state-of-the-art methods to ensure different notions of group fairness. This work offers the following key contributions: the provision of flexible, extensible, minimalistic, and research-oriented open-source code; the establishment of unified fairness method benchmarking pipelines; and extensive benchmarking, which yields key insights from 45,079 experiments, 14,428 GPU hours. We believe that our work will significantly facilitate the growth and development of the fairness research community.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 38 of its 62 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

38 of 62 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  7. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  8. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  9. 2014Certifying and removing disparate impactcited by 715
  10. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  11. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  12. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
26 more
  1. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  2. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  3. 2017Algorithmic decision making and the cost of fairnesscited by 402
  4. 2020Fairness in Machine Learning: A Surveycited by 369
  5. 2018Learning Adversarially Fair and Transferable Representationscited by 353
  6. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  7. 2015The Variational Fair Autoencodercited by 223
  8. 2015Censoring Representations with an Adversarycited by 191
  9. 2017Data Decisions and Theoretical Implications when Adversarially Learning Fair Representationscited by 165
  10. 2021A survey on datasets for fairness-aware machine learningcited by 157
  11. 2019Inherent Tradeoffs in Learning Fair Representationscited by 144
  12. 2019Wasserstein Fair Classificationcited by 131
  13. 2021Fair Mixup: Fairness via Interpolationcited by 109
  14. 2016Learning to Pivot with Adversarial Networkscited by 55
  15. 2022Fair and Optimal Classification via Post-Processingcited by 50
  16. 2019Average Individual Fairness: Algorithms, Generalization and Experimentscited by 46
  17. 2022Achieving Fairness at No Utility Cost via Data Reweighing with Influencecited by 38
  18. 2019R'enyi Fair Inferencecited by 33
  19. 2022Fairness Reprogrammingcited by 33
  20. 2022On Learning Fairness and Accuracy on Multiple Subgroupscited by 31
  21. 2022Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiencycited by 19
  22. 2022Fair Ranking with Noisy Protected Attributescited by 17
  23. 2019Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairnesscited by 16
  24. 2022Pushing the limits of fairness impossibility: Who's the fairest of them all?cited by 15
  25. 2022Optimal Transport of Classifiers to Fairnesscited by 12
  26. 2023Retiring DeltaDP: New Distribution-Level Metrics for Demographic Paritycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 11; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmarkcited by 28
  2. 2024Differentially Private Tabular Data Synthesis using Large Language Modelscited by 16
  3. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Promptscited by 15
  4. 2024OxonFair: A Flexible Toolkit for Algorithmic Fairnesscited by 12
  5. 2024CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Modelscited by 9
  6. 2024ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methodscited by 8
  7. 2023fairret: a Framework for Differentiable Fairness Regularization Termscited by 7
  8. 2023When Fairness Meets Privacy: Exploring Privacy Threats in Fair Binary Classifiers via Membership Inference Attackscited by 5
  9. 2025Revisiting Pre-processing Group Fairness: A Modular Benchmarking Frameworkcited by 4
  10. 2024On the Maximal Local Disparity of Fairness-Aware Classifierscited by 3
  11. 2024Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in MLcited by 3
  12. 2025Fairness Overfitting in Machine Learning: An Information-Theoretic Perspectivecited by 3
21 more
  1. 2023Towards Practical Non-Adversarial Distribution Matchingcited by 2
  2. 2024Learning Fairer Representations with FairVICcited by 2
  3. 2024Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challengescited by 2
  4. 2024Combining Machine Learning Defenses without Conflictscited by 2
  5. 2024Fairness via Independence: A (Conditional) Distance Covariance Frameworkcited by 2
  6. 2025Do Fairness Interventions Come at the Cost of Privacy: Evaluations for Binary Classifierscited by 2
  7. 2025TABFAIRGDT: A Fast Fair Tabular Data Generator using Autoregressive Decision Treescited by 2
  8. 2025Benchmarking Fairness-aware Graph Neural Networks in Knowledge Graphscited by 2
  9. 2025Bias Begins with Data: The FairGround Corpus for Robust and Reproducible Research on Algorithmic Fairnesscited by 2
  10. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Biascited by 1
  11. 2025LoGoFair: Post-Processing for Local and Global Fairness in Federated Learningcited by 1
  12. 2025Fair-FLIP: Fair Deepfake Detection with Fairness-Oriented Final Layer Input Prioritisingcited by 1
  13. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMscited by 1
  14. 2024On Demographic Group Fairness Guarantees in Deep Learningcited by 0
  15. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0
  16. 2025FedFACT: A Provable Framework for Controllable Group-Fairness Calibration in Federated Learningcited by 0
  17. 2025Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training of Deep Neural Networkscited by 0
  18. 2026PAC-Bayesian Generalization Guarantees for Fairness on Stochastic and Deterministic Classifierscited by 0
  19. 2026No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification modelscited by 0
  20. 2026A Model Ensemble-Based Post-Processing Framework for Fairness-Aware Predictioncited by 0
  21. 2026Demystifying the Optimal Fair Classifier in Multi-Class Classificationcited by 0