# A study on the distribution of social biases in self-supervised learning visual models

arXiv: https://arxiv.org/abs/2203.01854 · HTML: https://mlsanity.com/paper/2203.01854/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [Online Deep Clustering for Unsupervised Representation Learning](https://mlsanity.com/paper/2006.10645/) (2020)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination](https://mlsanity.com/paper/1408.6491/) (2014)
- [ClusterFit: Improving Generalization of Visual Representations](https://mlsanity.com/paper/1912.03330/) (2019)
- [Recovering from Biased Data: Can Fairness Constraints Improve Accuracy?](https://mlsanity.com/paper/1912.01094/) (2019)
- [Dual-Glance Model for Deciphering Social Relationships](https://mlsanity.com/paper/1708.00634/) (2017)

## Built on by (most cited first, 13 of 13)

- [Overwriting Pretrained Bias with Finetuning Data](https://mlsanity.com/paper/2303.06167/) (2023)
- [MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models](https://mlsanity.com/paper/2303.12734/) (2023)
- [Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models](https://mlsanity.com/paper/2304.13855/) (2023)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [A Multidimensional Analysis of Social Biases in Vision Transformers](https://mlsanity.com/paper/2308.01948/) (2023)
- [No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning](https://mlsanity.com/paper/2409.03025/) (2024)
- [Biased Attention: Do Vision Transformers Amplify Gender Bias More than Convolutional Neural Networks?](https://mlsanity.com/paper/2309.08760/) (2023)
- [A survey of Identification and mitigation of Machine Learning algorithmic biases in Image Analysis](https://mlsanity.com/paper/2210.04491/) (2022)
- [Zero-shot Visual Commonsense Immorality Prediction](https://mlsanity.com/paper/2211.05521/) (2022)
- [Mitigating Urban-Rural Disparities in Contrastive Representation Learning with Satellite Imagery](https://mlsanity.com/paper/2211.08672/) (2022)
- [QR-CLIP: Introducing Explicit Open-World Knowledge for Location and Time Reasoning](https://mlsanity.com/paper/2302.00952/) (2023)
- [A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models](https://mlsanity.com/paper/2505.20512/) (2025)
- [ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs](https://mlsanity.com/paper/2605.01971/) (2026)

## Abstract (arXiv)

Deep neural networks are efficient at learning the data distribution if it is sufficiently sampled. However, they can be strongly biased by non-relevant factors implicitly incorporated in the training data. These include operational biases, such as ineffective or uneven data sampling, but also ethical concerns, as the social biases are implicitly presenttextemdash even inadvertently, in the training data or explicitly defined in unfair training schedules. In tasks having impact on human processes, the learning of social biases may produce discriminatory, unethical and untrustworthy consequences. It is often assumed that social biases stem from supervised learning on labelled data, and thus, Self-Supervised Learning (SSL) wrongly appears as an efficient and bias-free solution, as it does not require labelled data. However, it was recently proven that a popular SSL method also incorporates biases. In this paper, we study the biases of a varied set of SSL visual models, trained using ImageNet data, using a method and dataset designed by psychological experts to measure social biases. We show that there is a correlation between the type of the SSL model and the number of biases that it incorporates. Furthermore, the results also suggest that this number does not strictly depend on the model's accuracy and changes throughout the network. Finally, we conclude that a careful SSL model selection process can reduce the number of social biases in the deployed model, whilst keeping high performance.