mlsanityData to 7 Oct 2026

Whitening Consistently Improves Self-Supervised Learning

arXiv 2408.07519 ↗First posted 14 Aug 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has been shown to be a powerful approach for learning visual representations. In this study, we propose incorporating ZCA whitening as the final layer of the encoder in self-supervised learning to enhance the quality of learned features by normalizing and decorrelating them. Although whitening has been utilized in SSL in previous works, its potential to universally improve any SSL model has not been explored. We demonstrate that adding whitening as the last layer of SSL pretrained encoders is independent of the self-supervised learning method and encoder architecture, thus it improves performance for a wide range of SSL methods across multiple encoder architectures and datasets. Our experiments show that whitening is capable of improving linear and k-NN probing accuracy by 1-5%. Additionally, we propose metrics that allow for a comprehensive analysis of the learned features, provide insights into the quality of the representations and help identify collapse patterns.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 14 Aug 2024.

Builds on

21 of 30 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
9 more
  1. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  2. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  3. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  4. 2020Whitening for Self-Supervised Representation Learningcited by 278
  5. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  6. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  7. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 92
  8. 2023The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Modelscited by 37
  9. 2024Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Modelscited by 2
  2. 2026Hypersolid: Emergent Vision Representations via Short-Range Repulsioncited by 1
  3. 2026Training-Free Fine-Grained Semantic Segmentations in Low Data Regimes: A FungiTastic Baselinecited by 1
  4. 2026Provable Subspace Identification of Nonlinear Multi-view CCAcited by 0
  5. 2026Self-Supervised Representation Learning via Hyperspherical Density Shapingcited by 0