mlsanityData to 14 Oct 2026

Self-Supervised Visual Representation Learning Using Lightweight Architectures

arXiv 2110.11160 ↗First posted 21 Oct 2021cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In self-supervised learning, a model is trained to solve a pretext task, using a data set whose annotations are created by a machine. The objective is to transfer the trained weights to perform a downstream task in the target domain. We critically examine the most notable pretext tasks to extract features from image data and further go on to conduct experiments on resource constrained networks, which aid faster experimentation and deployment. We study the performance of various self-supervised techniques keeping all other parameters uniform. We study the patterns that emerge by varying model type, size and amount of pre-training done for the backbone as well as establish a standard to compare against for future research. We also conduct comprehensive studies to understand the quality of representations learned by different architectures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 35 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

28 of 35 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
16 more
  1. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  4. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  5. 2016Colorful Image Colorizationcited by 1,114
  6. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  7. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  9. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  12. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  13. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 122
  14. 2017Representation Learning by Learning to Countcited by 120
  15. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  16. 2018Cross and Learn: Cross-Modal Self-Supervisioncited by 34

Built on by

Not cited yet.