mlsanityData to 7 Oct 2026

Evaluating the fairness of fine-tuning strategies in self-supervised learning

arXiv 2110.00538 ↗First posted 1 Oct 2021cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work we examine how fine-tuning impacts the fairness of contrastive Self-Supervised Learning (SSL) models. Our findings indicate that Batch Normalization (BN) statistics play a crucial role, and that updating only the BN statistics of a pre-trained SSL backbone improves its downstream fairness (36% worst subgroup, 25% mean subgroup gap). This procedure is competitive with supervised learning, while taking 4.4x less time to train and requiring only 0.35% as many parameters to be updated. Finally, inspired by recent work in supervised learning, we find that updating BN statistics and training residual skip connections (12.3% of the parameters) achieves parity with a fully fine-tuned model, while taking 1.33x less time to train.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

15 of 19 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2020Self-Supervised MultiModal Versatile Networkscited by 276
3 more
  1. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  2. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  3. 2020Fairness Metrics: A Comparative Analysiscited by 45

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 2.

3 papers.

  1. 2024Using Self-supervised Learning Can Improve Model Fairnesscited by 5
  2. 2022Achieving Fairness in Dermatological Disease Diagnosis through Automatic Weight Adjusting Federated Learning and Personalizationcited by 4
  3. 2024Evaluating Fairness in Self-supervised and Supervised Models for Sequential Datacited by 1