mlsanityData to 14 Oct 2026

Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks

arXiv 2111.11398 ↗First posted 22 Nov 2021cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning is a powerful paradigm for representation learning on unlabelled images. A wealth of effective new methods based on instance matching rely on data-augmentation to drive learning, and these have reached a rough agreement on an augmentation scheme that optimises popular recognition benchmarks. However, there is strong reason to suspect that different tasks in computer vision require features to encode different (in)variances, and therefore likely require different augmentation strategies. In this paper, we measure the invariances learned by contrastive methods and confirm that they do learn invariance to the augmentations used and further show that this invariance largely transfers to related real-world changes in pose and lighting. We show that learned invariances strongly affect downstream task performance and confirm that different downstream tasks benefit from polar opposite (in)variances, leading to performance loss when the standard augmentation strategy is used. Finally, we demonstrate that a simple fusion of representations with complementary invariances ensures wide transferability to all the diverse downstream tasks considered.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

31 of 51 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
19 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  6. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  7. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  8. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  9. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  10. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  11. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  12. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
  13. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  14. 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 142
  15. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  16. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 49
  17. 2017End-to-End Learning of Semantic Graspingcited by 20
  18. 2020Learning Translation Invariance in CNNscited by 8
  19. 2020Ventral-Dorsal Neural Networks: Object Detection via Selective Attentioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 7; 2024: 0; 2025: 0; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 250
  2. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  3. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  4. 2023Self-supervised Representation Learning From Random Data Projectorscited by 15
  5. 2022Data Augmentation is a Hyperparameter: Cherry-picked Self-Supervision for Unsupervised Anomaly Detection is Creating the Illusion of Successcited by 11
  6. 2022On the Limitations of General Purpose Domain Generalisation Methodscited by 10
  7. 2022TorMentor: Deterministic dynamic-path, data augmentations with fractalscited by 10
  8. 2022HyperNet: Self-Supervised Hyperspectral Spatial-Spectral Feature Understanding Network for Hyperspectral Change Detectioncited by 9
  9. 2023Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?cited by 5
  10. 2023Meta Omnium: A Benchmark for General-Purpose Learning-to-Learncited by 4
  11. 2022Local Manifold Augmentation for Multiview Semantic Consistencycited by 3
  12. 2023MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representationscited by 3
3 more
  1. 2026VIP-COP: Context Optimization for Tabular Foundation Modelscited by 3
  2. 2023Augmentation-aware Self-supervised Learning with Conditioned Projectorcited by 2
  3. 2026When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interferencecited by 0