# Self-Tuning for Data-Efficient Deep Learning

arXiv: https://arxiv.org/abs/2102.12903 · HTML: https://mlsanity.com/paper/2102.12903/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Analyzing the Performance of Multilayer Neural Networks for Object Recognition](https://mlsanity.com/paper/1407.1610/) (2014)
- [To Tune or Not To Tune? How About the Best of Both Worlds?](https://mlsanity.com/paper/1907.05338/) (2019)
- [Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer](https://mlsanity.com/paper/2004.05119/) (2020)

## Built on by (most cited first, 30 of 39)

- [Scribble-Supervised Medical Image Segmentation via Dual-Branch Network and Dynamically Mixed Pseudo Labels Supervision](https://mlsanity.com/paper/2203.02106/) (2022)
- [Debiased Self-Training for Semi-Supervised Learning](https://mlsanity.com/paper/2202.07136/) (2022)
- [Visual Recognition with Deep Nearest Centroids](https://mlsanity.com/paper/2209.07383/) (2022)
- [CYCLE: Learning to Self-Refine the Code Generation](https://mlsanity.com/paper/2403.18746/) (2024)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning](https://mlsanity.com/paper/2405.11756/) (2024)
- [Harnessing Hard Mixed Samples with Decoupled Regularizer](https://mlsanity.com/paper/2203.10761/) (2022)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Hyperspherical Consistency Regularization](https://mlsanity.com/paper/2206.00845/) (2022)
- [COSST: Multi-organ Segmentation with Partially Labeled Datasets Using Comprehensive Supervisions and Self-training](https://mlsanity.com/paper/2304.14030/) (2023)
- [Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning](https://mlsanity.com/paper/2209.07774/) (2022)
- [Fine-Grained Classification with Noisy Labels](https://mlsanity.com/paper/2303.02404/) (2023)
- [GraphTTA: Test Time Adaptation on Graph Neural Networks](https://mlsanity.com/paper/2208.09126/) (2022)
- [SemiReward: A General Reward Model for Semi-supervised Learning](https://mlsanity.com/paper/2310.03013/) (2023)
- [Improving Fine-Grained Visual Recognition in Low Data Regimes via Self-Boosting Attention Mechanism](https://mlsanity.com/paper/2208.00617/) (2022)
- [SIOD: Single Instance Annotated Per Category Per Image for Object Detection](https://mlsanity.com/paper/2203.15353/) (2022)
- [Improving Fine-tuning of Self-supervised Models with Contrastive Initialization](https://mlsanity.com/paper/2208.00238/) (2022)
- [From Big to Small: Adaptive Learning to Partial-Set Domains](https://mlsanity.com/paper/2203.07375/) (2022)
- [D3G: Exploring Gaussian Prior for Temporal Sentence Grounding with Glance Annotation](https://mlsanity.com/paper/2308.04197/) (2023)
- [Semi-supervised Drifted Stream Learning with Short Lookback](https://mlsanity.com/paper/2205.13066/) (2022)
- [Omni-Training: Bridging Pre-Training and Meta-Training for Few-Shot Learning](https://mlsanity.com/paper/2110.07510/) (2021)
- [Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime](https://mlsanity.com/paper/2205.03001/) (2022)
- [Language Semantic Graph Guided Data-Efficient Learning](https://mlsanity.com/paper/2311.08782/) (2023)
- [Test-Time Training for Semantic Segmentation with Output Contrastive Loss](https://mlsanity.com/paper/2311.07877/) (2023)
- [CE-SSL: Computation-Efficient Semi-Supervised Learning for ECG-based Cardiovascular Diseases Detection](https://mlsanity.com/paper/2406.14377/) (2024)
- [DeepTopoNet: A Framework for Subglacial Topography Estimation on the Greenland Ice Sheets](https://mlsanity.com/paper/2505.23980/) (2025)
- [TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Data](https://mlsanity.com/paper/2111.04798/) (2021)
- [Improving Compound Activity Classification via Deep Transfer and Representation Learning](https://mlsanity.com/paper/2111.07439/) (2021)
- [Decouple then Converge: Handling Unknown Unlabeled Distributions in Long-Tailed Semi-Supervised Learning](https://mlsanity.com/paper/2406.13187/) (2024)
- [A Re-node Self-training Approach for Deep Graph-based Semi-supervised Classification on Multi-view Image Data](https://mlsanity.com/paper/2510.24791/) (2025)

## Abstract (arXiv)

Deep learning has made revolutionary advances to diverse applications in the presence of large-scale labeled datasets. However, it is prohibitively time-costly and labor-expensive to collect sufficient labeled data in most realistic scenarios. To mitigate the requirement for labeled data, semi-supervised learning (SSL) focuses on simultaneously exploring both labeled and unlabeled data, while transfer learning (TL) popularizes a favorable practice of fine-tuning a pre-trained model to the target data. A dilemma is thus encountered: Without a decent pre-trained model to provide an implicit regularization, SSL through self-training from scratch will be easily misled by inaccurate pseudo-labels, especially in large-sized label space; Without exploring the intrinsic structure of unlabeled data, TL through fine-tuning from limited labeled data is at risk of under-transfer caused by model shift. To escape from this dilemma, we present Self-Tuning to enable data-efficient deep learning by unifying the exploration of labeled and unlabeled data and the transfer of a pre-trained model, as well as a Pseudo Group Contrast (PGC) mechanism to mitigate the reliance on pseudo-labels and boost the tolerance to false labels. Self-Tuning outperforms its SSL and TL counterparts on five tasks by sharp margins, e.g. it doubles the accuracy of fine-tuning on Cars with 15% labels.