mlsanityData to 7 Oct 2026

The Edge of Orthogonality: A Simple View of What Makes BYOL Tick

arXiv 2302.04817 ↗First posted 9 Feb 2023cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-predictive unsupervised learning methods such as BYOL or SimSiam have shown impressive results, and counter-intuitively, do not collapse to trivial representations. In this work, we aim at exploring the simplest possible mathematical arguments towards explaining the underlying mechanisms behind self-predictive unsupervised learning. We start with the observation that those methods crucially rely on the presence of a predictor network (and stop-gradient). With simple linear algebra, we show that when using a linear predictor, the optimal predictor is close to an orthogonal projection, and propose a general framework based on orthonormalization that enables to interpret and give intuition on why BYOL works. In addition, this framework demonstrates the crucial role of the exponential moving average and stop-gradient operator in BYOL as an efficient orthonormalization mechanism. We use these insights to propose four new closed-form predictor variants of BYOL to support our analysis. Our closed-form predictors outperform standard linear trainable predictor BYOL at 100 and 300 epochs (top-1 linear accuracy on ImageNet).

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

25 of 50 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2017Large Batch Training of Convolutional Networkscited by 529
13 more
  1. 2020Whitening for Self-Supervised Representation Learningcited by 278
  2. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  3. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  4. 2020BYOL works even without batch statisticscited by 95
  5. 2018SpectralNet: Spectral Clustering using Deep Neural Networkscited by 94
  6. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
  7. 2018Spectral Inference Networks: Unifying Deep and Spectral Learningcited by 34
  8. 2016Streaming PCA: Matching Matrix Bernstein and Near-Optimal Finite Sample Guarantees for Oja's Algorithmcited by 29
  9. 2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23
  10. 2022An Investigation into Whitening Loss for Self-supervised Learningcited by 21
  11. 2014A Stochastic PCA and SVD Algorithm with an Exponential Convergence Ratecited by 18
  12. 2022Neural Eigenfunctions Are Structured Representation Learnerscited by 16
  13. 2019Exponentially convergent stochastic k-PCA without variance reductioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 24
  2. 2023How to Scale Your EMAcited by 22
  3. 2023Graph-level Representation Learning with Joint-Embedding Predictive Architecturescited by 21
  4. 2025FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detectioncited by 10
  5. 2025On the Importance of Embedding Norms in Self-Supervised Learningcited by 9
  6. 2024The Hidden Pitfalls of the Cosine Similarity Losscited by 7
  7. 2025Representation Learning via Non-Contrastive Mutual Informationcited by 5
  8. 2024DiffClone: Enhanced Behaviour Cloning in Robotics with Diffusion-Driven Policy Learningcited by 3
  9. 2024On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learningcited by 3
  10. 2025Clustering via Self-Supervised Diffusioncited by 3
  11. 2025Semantic Concentration for Self-Supervised Dense Representations Learningcited by 3
  12. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
3 more
  1. 2025MatRL: Provably Generalizable Iterative Algorithm Discovery via Monte-Carlo Tree Searchcited by 1
  2. 2026Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learningcited by 0
  3. 2026Self-Supervised Learning with a Multi-Task Latent Space Objectivecited by 0