# Fairness Evaluation in Deepfake Detection Models using Metamorphic Testing

arXiv: https://arxiv.org/abs/2203.06825 · HTML: https://mlsanity.com/paper/2203.06825/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Face X-ray for More General Face Forgery Detection](https://mlsanity.com/paper/1912.13458/) (2019)
- [MesoNet: a Compact Facial Video Forgery Detection Network](https://mlsanity.com/paper/1809.00888/) (2018)
- [Adversarial Threats to DeepFake Detection: A Practical Perspective](https://mlsanity.com/paper/2011.09957/) (2020)
- [An Examination of Fairness of AI Models for Deepfake Detection](https://mlsanity.com/paper/2105.00558/) (2021)
- [Swapped Face Detection using Deep Learning and Subjective Assessment](https://mlsanity.com/paper/1909.04217/) (2019)
- [Robustness Evaluation of Stacked Generative Adversarial Networks using Metamorphic Testing](https://mlsanity.com/paper/2103.02870/) (2021)

## Built on by (most cited first, 16 of 16)

- [Preserving Fairness Generalization in Deepfake Detection](https://mlsanity.com/paper/2402.17229/) (2024)
- [Improving Fairness in Deepfake Detection](https://mlsanity.com/paper/2306.16635/) (2023)
- [Analyzing Fairness in Deepfake Detection With Massively Annotated Databases](https://mlsanity.com/paper/2208.05845/) (2022)
- [A Survey of Defenses Against AI-Generated Visual Media: Detection,Disruption, and Authentication](https://mlsanity.com/paper/2407.10575/) (2024)
- [FairSSD: Understanding Bias in Synthetic Speech Detectors](https://mlsanity.com/paper/2404.10989/) (2024)
- [Fair Deepfake Detectors Can Generalize](https://mlsanity.com/paper/2507.02645/) (2025)
- [Data-Driven Fairness Generalization for Deepfake Detection](https://mlsanity.com/paper/2412.16428/) (2024)
- [Gender Fairness in Audio Deepfake Detection: Performance and Disparity Analysis](https://mlsanity.com/paper/2603.09007/) (2026)
- [Robustness Evaluation in Hand Pose Estimation Models using Metamorphic Testing](https://mlsanity.com/paper/2303.04566/) (2023)
- [Redundant Semantic Environment Filling via Misleading-Learning for Fair Deepfake Detection](https://mlsanity.com/paper/2405.15173/) (2024)
- [Decoupling Bias, Aligning Distributions: Synergistic Fairness Optimization for Deepfake Detection](https://mlsanity.com/paper/2511.10150/) (2025)
- [Fair-FLIP: Fair Deepfake Detection with Fairness-Oriented Final Layer Input Prioritising](https://mlsanity.com/paper/2507.08912/) (2025)
- [Sensitive Region-based Metamorphic Testing Framework using Explainable AI](https://mlsanity.com/paper/2303.07580/) (2023)
- [Towards Exploring Fairness in Visual Transformer based Natural and GAN Image Detection Systems](https://mlsanity.com/paper/2310.12076/) (2023)
- [Wasserstein projection distance for fairness testing of regression models](https://mlsanity.com/paper/2510.04114/) (2025)
- [Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations](https://mlsanity.com/paper/2512.00556/) (2025)

## Abstract (arXiv)

Fairness of deepfake detectors in the presence of anomalies are not well investigated, especially if those anomalies are more prominent in either male or female subjects. The primary motivation for this work is to evaluate how deepfake detection model behaves under such anomalies. However, due to the black-box nature of deep learning (DL) and artificial intelligence (AI) systems, it is hard to predict the performance of a model when the input data is modified. Crucially, if this defect is not addressed properly, it will adversely affect the fairness of the model and result in discrimination of certain sub-population unintentionally. Therefore, the objective of this work is to adopt metamorphic testing to examine the reliability of the selected deepfake detection model, and how the transformation of input variation places influence on the output. We have chosen MesoInception-4, a state-of-the-art deepfake detection model, as the target model and makeup as the anomalies. Makeups are applied through utilizing the Dlib library to obtain the 68 facial landmarks prior to filling in the RGB values. Metamorphic relations are derived based on the notion that realistic perturbations of the input images, such as makeup, involving eyeliners, eyeshadows, blushes, and lipsticks (which are common cosmetic appearance) applied to male and female images, should not alter the output of the model by a huge margin. Furthermore, we narrow down the scope to focus on revealing potential gender biases in DL and AI systems. Specifically, we are interested to examine whether MesoInception-4 model produces unfair decisions, which should be considered as a consequence of robustness issues. The findings from our work have the potential to pave the way for new research directions in the quality assurance and fairness in DL and AI systems.