# Saliency-Based diversity and fairness Metric and FaceKeepOriginalAugment: A Novel Approach for Enhancing Fairness and Diversity

arXiv: https://arxiv.org/abs/2411.00831 · HTML: https://mlsanity.com/paper/2411.00831/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [BiaSwap: Removing dataset bias with bias-tailored swapping augmentation](https://mlsanity.com/paper/2108.10008/) (2021)
- [KeepAugment: A Simple Information-Preserving Data Augmentation Approach](https://mlsanity.com/paper/2011.11778/) (2020)
- [Hide-and-Seek: A Data Augmentation Technique for Weakly-Supervised Localization and Beyond](https://mlsanity.com/paper/1811.02545/) (2018)
- [Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasing](https://mlsanity.com/paper/2007.13632/) (2020)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [Image Data Augmentation Approaches: A Comprehensive Survey and Future directions](https://mlsanity.com/paper/2301.02830/) (2023)
- [Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models](https://mlsanity.com/paper/2304.13855/) (2023)
- [Towards A Holistic View of Bias in Machine Learning: Bridging Algorithmic Fairness and Imbalanced Learning](https://mlsanity.com/paper/2207.06084/) (2022)
- [Investigating Multi-Feature Selection and Ensembling for Audio Classification](https://mlsanity.com/paper/2206.07511/) (2022)
- [Random Data Augmentation based Enhancement: A Generalized Enhancement Approach for Medical Datasets](https://mlsanity.com/paper/2210.00824/) (2022)
- [Understanding EEG signals for subject-wise Definition of Armoni Activities](https://mlsanity.com/paper/2301.00948/) (2023)
- [Gender Bias in Multimodal Models: A Transnational Feminist Approach Considering Geographical Region and Culture](https://mlsanity.com/paper/2309.04997/) (2023)
- [AudRandAug: Random Image Augmentations for Audio Classification](https://mlsanity.com/paper/2309.04762/) (2023)
- [KeepOriginalAugment: Single Image-based Better Information-Preserving Data Augmentation Approach](https://mlsanity.com/paper/2405.06354/) (2024)

## Abstract (arXiv)

Data augmentation has become a pivotal tool in enhancing the performance of computer vision tasks, with the KeepOriginalAugment method emerging as a standout technique for its intelligent incorporation of salient regions within less prominent areas, enabling augmentation in both regions. Despite its success in image classification, its potential in addressing biases remains unexplored. In this study, we introduce an extension of the KeepOriginalAugment method, termed FaceKeepOriginalAugment, which explores various debiasing aspects-geographical, gender, and stereotypical biases-in computer vision models. By maintaining a delicate balance between data diversity and information preservation, our approach empowers models to exploit both diverse salient and non-salient regions, thereby fostering increased diversity and debiasing effects. We investigate multiple strategies for determining the placement of the salient region and swapping perspectives to decide which part undergoes augmentation. Leveraging the Image Similarity Score (ISS), we quantify dataset diversity across a range of datasets, including Flickr Faces HQ (FFHQ), WIKI, IMDB, Labelled Faces in the Wild (LFW), UTK Faces, and Diverse Dataset. We evaluate the effectiveness of FaceKeepOriginalAugment in mitigating gender bias across CEO, Engineer, Nurse, and School Teacher datasets, utilizing the Image-Image Association Score (IIAS) in convolutional neural networks (CNNs) and vision transformers (ViTs). Our findings shows the efficacy of FaceKeepOriginalAugment in promoting fairness and inclusivity within computer vision models, demonstrated by reduced gender bias and enhanced overall fairness. Additionally, we introduce a novel metric, Saliency-Based Diversity and Fairness Metric, which quantifies both diversity and fairness while handling data imbalance across various datasets.