# Using Self-supervised Learning Can Improve Model Fairness

arXiv: https://arxiv.org/abs/2406.02361 · HTML: https://mlsanity.com/paper/2406.02361/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 67 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 33)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [Why Is My Classifier Discriminatory?](https://mlsanity.com/paper/1805.12002/) (2018)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [How Useful is Self-Supervised Pretraining for Visual Tasks?](https://mlsanity.com/paper/2003.14323/) (2020)
- [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Fairness Metrics: A Comparative Analysis](https://mlsanity.com/paper/2001.07864/) (2020)
- [Neighborhood Contrastive Learning Applied to Online Patient Monitoring](https://mlsanity.com/paper/2106.05142/) (2021)
- [Overwriting Pretrained Bias with Finetuning Data](https://mlsanity.com/paper/2303.06167/) (2023)
- [SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data](https://mlsanity.com/paper/2102.06073/) (2021)
- [GLOBEM Dataset: Multi-Year Datasets for Longitudinal Human Behavior Modeling Generalization](https://mlsanity.com/paper/2211.02733/) (2022)
- [Self-supervised transfer learning of physiological representations from free-living wearable data](https://mlsanity.com/paper/2011.12121/) (2020)
- [Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning](https://mlsanity.com/paper/2106.02866/) (2021)
- [Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?](https://mlsanity.com/paper/2110.00528/) (2021)
- [Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks](https://mlsanity.com/paper/2304.03935/) (2023)
- [Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows](https://mlsanity.com/paper/2201.07677/) (2022)
- [(Un)fair devices: Moving beyond AI accuracy in personal sensing](https://mlsanity.com/paper/2303.15585/) (2023)

## Built on by (most cited first, 5 of 5)

- [Semi-Supervised Risk Control via Prediction-Powered Inference](https://mlsanity.com/paper/2412.11174/) (2024)
- [Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impact](https://mlsanity.com/paper/2412.19124/) (2024)
- [Task-Agnostic Attacks Against Vision Foundation Models](https://mlsanity.com/paper/2503.03842/) (2025)
- [Adversarial Robustness of Discriminative Self-Supervised Learning in Vision](https://mlsanity.com/paper/2503.06361/) (2025)
- [FairSSL: Fair Multimodal Self-Supervised Learning](https://mlsanity.com/paper/2508.16748/) (2025)

## Abstract (arXiv)

Self-supervised learning (SSL) has become the de facto training paradigm of large models, where pre-training is followed by supervised fine-tuning using domain-specific data and labels. Despite demonstrating comparable performance with supervised methods, comprehensive efforts to assess SSL's impact on machine learning fairness (i.e., performing equally on different demographic breakdowns) are lacking. Hypothesizing that SSL models would learn more generic, hence less biased representations, this study explores the impact of pre-training and fine-tuning strategies on fairness. We introduce a fairness assessment framework for SSL, comprising five stages: defining dataset requirements, pre-training, fine-tuning with gradual unfreezing, assessing representation similarity conditioned on demographics, and establishing domain-specific evaluation processes. We evaluate our method's generalizability on three real-world human-centric datasets (i.e., MIMIC, MESA, and GLOBEM) by systematically comparing hundreds of SSL and fine-tuned models on various dimensions spanning from the intermediate representations to appropriate evaluation metrics. Our findings demonstrate that SSL can significantly improve model fairness, while maintaining performance on par with supervised methods-exhibiting up to a 30% increase in fairness with minimal loss in performance through self-supervision. We posit that such differences can be attributed to representation dissimilarities found between the best- and the worst-performing demographics across models-up to x13 greater for protected attributes with larger performance discrepancies between segments.