# Unsupervised Finetuning

arXiv: https://arxiv.org/abs/2110.09510 · HTML: https://mlsanity.com/paper/2110.09510/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Learning Discriminative Features with Multiple Granularities for Person Re-Identification](https://mlsanity.com/paper/1804.01438/) (2018)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [Intriguing Properties of Contrastive Losses](https://mlsanity.com/paper/2011.02803/) (2020)
- [Unsupervised Pre-training for Person Re-identification](https://mlsanity.com/paper/2012.03753/) (2020)
- [Self-training for Few-shot Transfer Across Extreme Task Differences](https://mlsanity.com/paper/2010.07734/) (2020)
- [Improve Unsupervised Pretraining for Few-label Transfer](https://mlsanity.com/paper/2107.12369/) (2021)
- [Improving Person Re-identification with Iterative Impression Aggregation](https://mlsanity.com/paper/2009.10066/) (2020)

## Built on by (most cited first, 6 of 6)

- [Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework](https://mlsanity.com/paper/2112.05141/) (2021)
- [COFT-AD: COntrastive Fine-Tuning for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2402.18998/) (2024)
- [Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime](https://mlsanity.com/paper/2205.03001/) (2022)
- [InsightNet: Structured Insight Mining from Customer Feedback](https://mlsanity.com/paper/2405.07195/) (2024)
- [Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2112.12522/) (2021)
- [Efficient Multi-Prize Lottery Tickets: Enhanced Accuracy, Training, and Inference Speed](https://mlsanity.com/paper/2209.12839/) (2022)

## Abstract (arXiv)

This paper studies "unsupervised finetuning", the symmetrical problem of the well-known "supervised finetuning". Given a pretrained model and small-scale unlabeled target data, unsupervised finetuning is to adapt the representation pretrained from the source domain to the target domain so that better transfer performance can be obtained. This problem is more challenging than the supervised counterpart, as the low data density in the small-scale target data is not friendly for unsupervised learning, leading to the damage of the pretrained representation and poor representation in the target domain. In this paper, we find the source data is crucial when shifting the finetuning paradigm from supervise to unsupervise, and propose two simple and effective strategies to combine source and target data into unsupervised finetuning: "sparse source data replaying", and "data mixing". The motivation of the former strategy is to add a small portion of source data back to occupy their pretrained representation space and help push the target data to reside in a smaller compact space; and the motivation of the latter strategy is to increase the data density and help learn more compact representation. To demonstrate the effectiveness of our proposed ``unsupervised finetuning'' strategy, we conduct extensive experiments on multiple different target datasets, which show better transfer performance than the naive strategy.