# Bi-tuning of Pre-trained Representations

arXiv: https://arxiv.org/abs/2011.06182 · HTML: https://mlsanity.com/paper/2011.06182/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 15 of 15)

- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning](https://mlsanity.com/paper/2102.06605/) (2021)
- [Zoo-Tuning: Adaptive Transfer from a Zoo of Models](https://mlsanity.com/paper/2106.15434/) (2021)
- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [Hyperspherical Consistency Regularization](https://mlsanity.com/paper/2206.00845/) (2022)
- [Improving Fine-tuning of Self-supervised Models with Contrastive Initialization](https://mlsanity.com/paper/2208.00238/) (2022)
- [Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning](https://mlsanity.com/paper/2108.02959/) (2021)
- [Neighborhood Consensus Contrastive Learning for Backward-Compatible Representation](https://mlsanity.com/paper/2108.03372/) (2021)
- [DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration](https://mlsanity.com/paper/2308.12058/) (2023)
- [Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models](https://mlsanity.com/paper/2206.03726/) (2022)
- [Rethinking Two Consensuses of the Transferability in Deep Learning](https://mlsanity.com/paper/2212.00399/) (2022)
- [Adversarial Lagrangian Integrated Contrastive Embedding for Limited Size Datasets](https://mlsanity.com/paper/2210.03261/) (2022)
- [Delta-Patching: A Framework for Rapid Adaptation of Pre-trained Convolutional Networks without Base Performance Loss](https://mlsanity.com/paper/2303.14772/) (2023)
- [Concept-wise Fine-tuning Matters in Preventing Negative Transfer](https://mlsanity.com/paper/2311.06868/) (2023)
- [Finetuning Pre-trained Model with Limited Data for LiDAR-based 3D Object Detection by Bridging Domain Gaps](https://mlsanity.com/paper/2410.01319/) (2024)

## Abstract (arXiv)

It is common within the deep learning community to first pre-train a deep neural network from a large-scale dataset and then fine-tune the pre-trained model to a specific downstream task. Recently, both supervised and unsupervised pre-training approaches to learning representations have achieved remarkable advances, which exploit the discriminative knowledge of labels and the intrinsic structure of data, respectively. It follows natural intuition that both discriminative knowledge and intrinsic structure of the downstream task can be useful for fine-tuning, however, existing fine-tuning methods mainly leverage the former and discard the latter. A question arises: How to fully explore the intrinsic structure of data for boosting fine-tuning? In this paper, we propose Bi-tuning, a general learning framework to fine-tuning both supervised and unsupervised pre-trained representations to downstream tasks. Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the backbone of pre-trained representations: a classifier head with an improved contrastive cross-entropy loss to better leverage the label information in an instance-contrast way, and a projector head with a newly-designed categorical contrastive learning loss to fully exploit the intrinsic structure of data in a category-consistent way. Comprehensive experiments confirm that Bi-tuning achieves state-of-the-art results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins (e.g. 10.7% absolute rise in accuracy on CUB in low-data regime).