mlsanityData to 14 Oct 2026

Modulate Your Spectrum in Self-Supervised Learning

arXiv 2305.16789 ↗First posted 26 May 2023cs.LG cs.CV eess.SPcited by 9

Not on the map. Why?

Abstract (arXiv)

Whitening loss offers a theoretical guarantee against feature collapse in self-supervised learning (SSL) with joint embedding architectures. Typically, it involves a hard whitening approach, transforming the embedding and applying loss to the whitened output. In this work, we introduce Spectral Transformation (ST), a framework to modulate the spectrum of embedding and to seek for functions beyond whitening that can avoid dimensional collapse. We show that whitening is a special instance of ST by definition, and our empirical investigations unveil other ST instances capable of preventing collapse. Additionally, we propose a novel ST instance named IterNorm with trace loss (INTL). Theoretical analysis confirms INTL's efficacy in preventing collapse and modulating the spectrum of embedding toward equal-eigenvalues during optimization. Our experiments on ImageNet classification and COCO object detection demonstrate INTL's potential in learning superior representations. The code is available at https://github.com/winci-ai/INTL.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

44 of 52 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
32 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2019Contrastive Multiview Codingcited by 1,301
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  8. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  9. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  12. 2021Graph Self-Supervised Learning: A Surveycited by 289
  13. 2020Whitening for Self-Supervised Representation Learningcited by 279
  14. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  15. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  16. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  17. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 141
  18. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  19. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  20. 2021Contrastive Learning with Stronger Augmentationscited by 100
  21. 2020BYOL works even without batch statisticscited by 95
  22. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  23. 2018Decorrelated Batch Normalizationcited by 83
  24. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  25. 2021Self-supervised Video Transformercited by 64
  26. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  27. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  28. 2022Spectral Feature Augmentation for Graph Contrastive Learning and Beyondcited by 49
  29. 2018Whitening and Coloring batch transform for GANscited by 27
  30. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  31. 2022Investigating Power laws in Deep Representation Learningcited by 17
  32. 2019Network Deconvolutioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 1.

9 papers.

  1. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
  2. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  3. 2025Clustering Properties of Self-Supervised Learningcited by 8
  4. 2024Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalancecited by 2
  5. 2025T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learningcited by 2
  6. 2025Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priorscited by 1
  7. 2025IBNorm: Information-Bottleneck Inspired Normalization for Representation Learningcited by 1
  8. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  9. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Modelscited by 0