# Fairness Testing: A Comprehensive Survey and Analysis of Trends

arXiv: https://arxiv.org/abs/2207.10223 · HTML: https://mlsanity.com/paper/2207.10223/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2022.

## Built on by (most cited first, 30 of 62)

- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges](https://mlsanity.com/paper/2305.06969/) (2023)
- [Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testing](https://mlsanity.com/paper/2305.11602/) (2023)
- [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://mlsanity.com/paper/2401.00763/) (2024)
- [Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking](https://mlsanity.com/paper/2302.08018/) (2023)
- [Fairness Testing of Large Language Models in Role-Playing](https://mlsanity.com/paper/2411.00585/) (2024)
- ["Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Process](https://mlsanity.com/paper/2305.02626/) (2023)
- [An AI System Evaluation Framework for Advancing AI Safety: Terminology, Taxonomy, Lifecycle Mapping](https://mlsanity.com/paper/2404.05388/) (2024)
- [Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2308.02935/) (2023)
- [A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models](https://mlsanity.com/paper/2401.03695/) (2024)
- [FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering](https://mlsanity.com/paper/2505.19562/) (2025)
- [Implications of the AI Act for Non-Discrimination Law and Algorithmic Fairness](https://mlsanity.com/paper/2403.20089/) (2024)
- [FairZK: A Scalable System to Prove Machine Learning Fairness in Zero-Knowledge](https://mlsanity.com/paper/2505.07997/) (2025)
- [Fairness Concerns in App Reviews: A Study on AI-based Mobile Apps](https://mlsanity.com/paper/2401.08097/) (2024)
- [A Comprehensive View of the Biases of Toxicity and Sentiment Analysis Methods Towards Utterances with African American English Expressions](https://mlsanity.com/paper/2401.12720/) (2024)
- [Do Existing Testing Tools Really Uncover Gender Bias in Text-to-Image Models?](https://mlsanity.com/paper/2501.15775/) (2025)
- [FairRec: Fairness Testing for Deep Recommender Systems](https://mlsanity.com/paper/2304.07030/) (2023)
- [RAI4IoE: Responsible AI for Enabling the Internet of Energy](https://mlsanity.com/paper/2309.11691/) (2023)
- [Diversity Drives Fairness: Ensemble of Higher Order Mutants for Intersectional Fairness of Machine Learning Software](https://mlsanity.com/paper/2412.08167/) (2024)
- [Generative AI for Testing of Autonomous Driving Systems: A Survey](https://mlsanity.com/paper/2508.19882/) (2025)
- [Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances](https://mlsanity.com/paper/2209.08321/) (2022)
- [Mutatis Mutandis: Revisiting the Comparator in Discrimination Testing](https://mlsanity.com/paper/2405.13693/) (2024)
- [Counterfactual Situation Testing: From Single to Multidimensional Discrimination](https://mlsanity.com/paper/2502.01267/) (2025)
- [Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages](https://mlsanity.com/paper/2509.13803/) (2025)
- [A Comprehensive Study of Deep Learning Model Fixing Approaches](https://mlsanity.com/paper/2512.23745/) (2025)
- [An Operational Perspective to Fairness Interventions: Where and How to Intervene](https://mlsanity.com/paper/2302.01574/) (2023)
- [Towards Better Fairness-Utility Trade-off: A Comprehensive Measurement-Based Reinforcement Learning Framework](https://mlsanity.com/paper/2307.11379/) (2023)
- [Mapping the Potential of Explainable AI for Fairness Along the AI Lifecycle](https://mlsanity.com/paper/2404.18736/) (2024)

## Abstract (arXiv)

Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this paper offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely-adopted datasets and open-source tools for fairness testing.