mlsanityData to 14 Oct 2026

Improving Fine-tuning of Self-supervised Models with Contrastive Initialization

arXiv 2208.00238 ↗First posted 30 Jul 2022cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has achieved remarkable performance in pretraining the models that can be further used in downstream tasks via fine-tuning. However, these self-supervised models may not capture meaningful semantic information since the images belonging to the same class are always regarded as negative pairs in the contrastive loss. Consequently, the images of the same class are often located far away from each other in learned feature space, which would inevitably hamper the fine-tuning process. To address this issue, we seek to provide a better initialization for the self-supervised models by enhancing the semantic information. To this end, we propose a Contrastive Initialization (COIN) method that breaks the standard fine-tuning pipeline by introducing an extra initialization stage before fine-tuning. Extensive experiments show that, with the enriched semantics, our COIN significantly outperforms existing methods without introducing extra training cost and sets new state-of-the-arts on multiple downstream tasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 54 of its 78 references that are in the map. First posted on arXiv 30 Jul 2022.

Builds on

54 of 78 references.

References · 54

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
38 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  2. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  3. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  4. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  5. 2021Domain Generalization: A Surveycited by 764
  6. 2018Deep Visual Domain Adaptation: A Surveycited by 635
  7. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  8. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  9. 2020Understanding the Behaviour of Contrastive Losscited by 398
  10. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 341
  11. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  12. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
  13. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  14. 2020Intriguing Properties of Contrastive Lossescited by 130
  15. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  16. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  17. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  18. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  19. 2020Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolutioncited by 90
  20. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  21. 2019DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networkscited by 80
  22. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  23. 2021Equivariant Contrastive Learningcited by 66
  24. 2021Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuningcited by 50
  25. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  26. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  27. 2021Robust Contrastive Learning Using Negative Samples with Diminished Semanticscited by 43
  28. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  29. 2021Self-Tuning for Data-Efficient Deep Learningcited by 39
  30. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37
  31. 2019NAT: Neural Architecture Transformer for Accurate and Compact Architecturescited by 33
  32. 2020RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayErcited by 27
  33. 2022Enhancing Hyperbolic Graph Embeddings via Contrastive Learningcited by 20
  34. 2017Mix-and-Match Tuning for Self-Supervised Semantic Segmentationcited by 17
  35. 2021Towards Accurate and Compact Architectures via Neural Architecture Transformercited by 17
  36. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  37. 2020Bi-tuning of Pre-trained Representationscited by 15
  38. 2022Improving Robustness by Enhancing Weak Subnetscited by 13

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 1; 2026: 1.

10 papers.

  1. 2023Robustifying Token Attention for Vision Transformerscited by 19
  2. 2022Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Imagescited by 8
  3. 2024How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignmentcited by 6
  4. 2022Boosting Semi-Supervised Learning with Contrastive Complementary Labelingcited by 5
  5. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  6. 2023Controlled Randomness Improves the Performance of Transformer Modelscited by 0
  7. 2023Concept-wise Fine-tuning Matters in Preventing Negative Transfercited by 0
  8. 2024Finetuning Pre-trained Model with Limited Data for LiDAR-based 3D Object Detection by Bridging Domain Gapscited by 0
  9. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  10. 2026A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learningcited by 0