mlsanityData to 7 Oct 2026

Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learning

arXiv 2601.20154 ↗First posted 28 Jan 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has improved empirical performance by unleashing the power of unlabeled data for practical applications. Specifically, SSL extracts the representation from massive unlabeled data, which will be transferred to a plenty of down streaming tasks with limited data. The significant improvement on diverse applications of representation learning has attracted increasing attention, resulting in a variety of dramatically different self-supervised learning objectives for representation extraction, with an assortment of learning procedures, but the lack of a clear and unified understanding. Such an absence hampers the ongoing development of representation learning, leaving a theoretical understanding missing, principles for efficient algorithm design unclear, and the use of representation learning methods in practice unjustified. The urgency for a unified framework is further motivated by the rapid growth in representation learning methods. In this paper, we are therefore compelled to develop a principled foundation of representation learning. We first theoretically investigate the sufficiency of the representation from a spectral representation view, which reveals the spectral essence of the existing successful SSL algorithms and paves the path to a unified framework for understanding and analysis. Such a framework work also inspires the development of more efficient and easy-to-use representation learning algorithms with principled way in real-world applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 64 of its 97 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

64 of 97 references.

References · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2025DINOv3cited by 1,297
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  8. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  10. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  11. 2019On Mutual Information Maximization for Representation Learningcited by 315
  12. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  13. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  14. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  15. 2023A Cookbook of Self-Supervised Learningcited by 248
  16. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  17. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  18. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  19. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  20. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  21. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 109
  22. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  23. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  24. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  25. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  26. 2022Audio Self-supervised Learning: A Surveycited by 65
  27. 2022Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Seriescited by 54
  28. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  29. 2022Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performancecited by 40
  30. 2014Scalable Kernel Methods via Doubly Stochastic Gradientscited by 36
  31. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  32. 2019Exponential Family Estimation via Adversarial Dynamics Embeddingcited by 30
  33. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  34. 2021A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learningcited by 24
  35. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  36. 2015Nonparametric Canonical Correlation Analysiscited by 22
  37. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  38. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15

50 most cited of 64 All 64 (JSON)

Built on by

Not cited yet.