Self-Labeling Refinement for Robust Representation Learning with Bootstrap Your Own Latent
Not on the map. Why?
Abstract (arXiv)
In this work, we have worked towards two major goals. Firstly, we have investigated the importance of Batch Normalisation (BN) layers in a non-contrastive representation learning framework called Bootstrap Your Own Latent (BYOL). We conducted several experiments to conclude that BN layers are not necessary for representation learning in BYOL. Moreover, BYOL only learns from the positive pairs of images but ignores other semantically similar images in the same input batch. For the second goal, we have introduced two new loss functions to determine the semantically similar pairs in the same input batch of images and reduce the distance between their representations. These loss functions are Cross-Cosine Similarity Loss (CCSL) and Cross-Sigmoid Similarity Loss (CSSL). Using the proposed loss functions, we are able to surpass the performance of Vanilla BYOL (71.04%) by training the BYOL framework using CCSL loss (76.87%) on the STL10 dataset. BYOL trained using CSSL loss performs comparably with Vanilla BYOL.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 9 Apr 2022.
Builds on
33 of 44 references.
References · 33
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Layer Normalizationcited by 5,579
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
- 2018Group Normalizationcited by 1,514
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
21 more
- 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
- 2019Contrastive Multiview Codingcited by 1,297
- 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
- 2016Colorful Image Colorizationcited by 1,113
- 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
- 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
- 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
- 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
- 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
- 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
- 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
- 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
- 2016Learning Representations for Automatic Colorizationcited by 251
- 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
- 2017Multi-task Self-Supervised Visual Learningcited by 200
- 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
- 2020BYOL works even without batch statisticscited by 95
- 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
- 2019Micro-Batch Training with Batch-Channel Normalization and Weight Standardizationcited by 45
- 2020Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approachcited by 12
- 2021A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learningcited by 7
Built on by
Citing papers per year: 2025: 1.
1 paper.