mlsanityData to 14 Oct 2026

Revisiting the Critical Factors of Augmentation-Invariant Representation Learning

arXiv 2208.00275 ↗First posted 30 Jul 2022cs.CV cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We focus on better understanding the critical factors of augmentation-invariant representation learning. We revisit MoCo v2 and BYOL and try to prove the authenticity of the following assumption: different frameworks bring about representations of different characteristics even with the same pretext task. We establish the first benchmark for fair comparisons between MoCo v2 and BYOL, and observe: (i) sophisticated model configurations enable better adaptation to pre-training dataset; (ii) mismatched optimization strategies of pre-training and fine-tuning hinder model from achieving competitive transfer performances. Given the fair benchmark, we make further investigation and find asymmetry of network structure endows contrastive frameworks to work well under the linear evaluation protocol, while may hurt the transfer performances on long-tailed classification tasks. Moreover, negative samples do not make models more sensible to the choice of data augmentations, nor does the asymmetric network structure. We believe our findings provide useful information for future work.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 44 of its 50 references that are in the map. First posted on arXiv 30 Jul 2022.

Builds on

44 of 50 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2017Mask R-CNNcited by 7,277
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,866
32 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  3. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  13. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  14. 2019Contrastive Multiview Codingcited by 1,301
  15. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  16. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  17. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  18. 2017Large Batch Training of Convolutional Networkscited by 535
  19. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  20. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  21. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  22. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  23. 2020Whitening for Self-Supervised Representation Learningcited by 279
  24. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  25. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  26. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  27. 2016What makes ImageNet good for transfer learning?cited by 192
  28. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  29. 2020What makes instance discrimination good for transfer learning?cited by 111
  30. 2020BYOL works even without batch statisticscited by 95
  31. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  32. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 0; 2025: 1; 2026: 2.

6 papers.

  1. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  2. 2022Pixel-Wise Contrastive Distillationcited by 5
  3. 2023From Pretext to Purpose: Batch-Adaptive Self-Supervised Learningcited by 1
  4. 2025Scaling Non-Parametric Sampling with Representationcited by 0
  5. 2026Maximizing Incremental Information Entropy for Contrastive Learningcited by 0
  6. 2026Image Augmentation as Test Generation for Deep Learning-Based Image Retrieval Systemscited by 0