Data to 14 Oct 2026

Fairness Testing: A Comprehensive Survey and Analysis of Trends

arXiv 2207.10223 ↗First posted 20 Jul 2022cs.SEcited by 62

Not on the map. Why?

Abstract (arXiv)

Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this paper offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely-adopted datasets and open-source tools for fairness testing.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Jul 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 17; 2025: 23; 2026: 9.

62 papers.

Sort:

By area:

2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
2023A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challengescited by 41
2023Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testingcited by 23
2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 16
2023Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinkingcited by 14
2024Fairness Testing of Large Language Models in Role-Playingcited by 11
2023"Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Processcited by 9
2024An AI System Evaluation Framework for Advancing AI Safety: Terminology, Taxonomy, Lifecycle Mappingcited by 8
2023Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systemscited by 7
2024A Large-Scale Empirical Study on Improving the Fairness of Image Classification Modelscited by 6

38 more

2025FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answeringcited by 6
2024Implications of the AI Act for Non-Discrimination Law and Algorithmic Fairnesscited by 5
2025FairZK: A Scalable System to Prove Machine Learning Fairness in Zero-Knowledgecited by 5
2024Fairness Concerns in App Reviews: A Study on AI-based Mobile Appscited by 4
2024A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressionscited by 4
2025Do Existing Testing Tools Really Uncover Gender Bias in Text-to-Image Models?cited by 4
2023FairRec: Fairness Testing for Deep Recommender Systemscited by 3
2023RAI4IoE: Responsible AI for Enabling the Internet of Energycited by 3
2024Diversity Drives Fairness: Ensemble of Higher Order Mutants for Intersectional Fairness of Machine Learning Softwarecited by 3
2025Generative AI for Testing of Autonomous Driving Systems: A Surveycited by 3
2022Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instancescited by 2
2024Mutatis Mutandis: Revisiting the Comparator in Discrimination Testingcited by 2
2025Counterfactual Situation Testing: From Single to Multidimensional Discriminationcited by 2
2025Measuring Gender Bias in Job Title Matching for Grammatical Gender Languagescited by 2
2025A Comprehensive Study of Deep Learning Model Fixing Approachescited by 2
2023An Operational Perspective to Fairness Interventions: Where and How to Intervenecited by 1
2023Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Frameworkcited by 1
2024Mapping the Potential of Explainable AI for Fairness Along the AI Lifecyclecited by 1
2024Unbiasing on the Fly: Explanation-Guided Human Oversight of Machine Learning System Decisionscited by 1
2024FAIREDU: A Multiple Regression-Based Method for Enhancing Fairness in Machine Learning Models for Educational Applicationscited by 1
2025Software Engineering Principles for Fairer Systems: Experiments with GroupCARTcited by 1
2025Intersectional Divergence: Measuring Fairness in Regressioncited by 1
2025Fairness-aware Anomaly Detection via Fair Projectioncited by 1
2025Software Fairness Dilemma: Is Bias Mitigation a Zero-Sum Game?cited by 1
2025Beyond Overall Accuracy: Pose- and Occlusion-driven Fairness Analysis in Pedestrian Detection for Autonomous Drivingcited by 1
2025Fairness Testing in Retrieval-Augmented Generation: How Small Perturbations Reveal Bias in Small Language Modelscited by 1
2025Causally Perturbed Fairness Testingcited by 1
2023Distribution-aware Fairness Test Generationcited by 0
2023Fairness Improvement with Multiple Protected Attributes: How Far Are We?cited by 0
2023Search-Based Fairness Testing: An Overviewcited by 0
2024Fairpriori: Improving Biased Subgroup Discovery for Deep Neural Network Fairnesscited by 0
2024Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning Systemscited by 0
2025Automated Harmfulness Testing for Code Large Language Modelscited by 0
2025Whence Is A Model Fair? Fixing Fairness Bugs via Propensity Score Matchingcited by 0
2025Testing Individual Fairness in Graph Neural Networkscited by 0
2025Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training of Deep Neural Networkscited by 0
2025FairFLRep: Fairness aware fault localization and repair of Deep Neural Networkscited by 0
2025CAF'E: Causal Black-Box Testing of Machine Unlearningcited by 0

50 most cited of 62 All 62 (JSON)