# Adaptive Consistency Regularization for Semi-Supervised Transfer Learning

arXiv: https://arxiv.org/abs/2103.02193 · HTML: https://mlsanity.com/paper/2103.02193/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning](https://mlsanity.com/paper/1702.08690/) (2017)
- [TransMatch: A Transfer-Learning Scheme for Semi-Supervised Few-Shot Learning](https://mlsanity.com/paper/1912.09033/) (2019)
- [Alleviating Semantic-level Shift: A Semi-supervised Domain Adaptation Method for Semantic Segmentation](https://mlsanity.com/paper/2004.00794/) (2020)
- [Domain Adaptive Transfer Learning with Specialist Models](https://mlsanity.com/paper/1811.07056/) (2018)
- [RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr](https://mlsanity.com/paper/2007.03349/) (2020)
- [MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs](https://mlsanity.com/paper/1712.06957/) (2017)
- [When Semi-Supervised Learning Meets Transfer Learning: Training Strategies, Models and Datasets](https://mlsanity.com/paper/1812.05313/) (2018)

## Built on by (most cited first, 30 of 37)

- [Test-Time Model Adaptation with Only Forward Passes](https://mlsanity.com/paper/2404.01650/) (2024)
- [A Closer Look at Few-shot Image Generation](https://mlsanity.com/paper/2205.03805/) (2022)
- [Semi-Supervised Domain Generalization with Stochastic StyleMatch](https://mlsanity.com/paper/2106.00592/) (2021)
- [Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students](https://mlsanity.com/paper/2209.02178/) (2022)
- [InstanT: Semi-supervised Learning with Instance-dependent Thresholds](https://mlsanity.com/paper/2310.18910/) (2023)
- [Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment](https://mlsanity.com/paper/2304.14623/) (2023)
- [Towards Generalizing to Unseen Domains with Few Labels](https://mlsanity.com/paper/2403.11674/) (2024)
- [Pushing the Performance Limit of Scene Text Recognizer without Human Annotation](https://mlsanity.com/paper/2204.07714/) (2022)
- [Knowledge Sharing via Domain Adaptation in Customs Fraud Detection](https://mlsanity.com/paper/2201.06759/) (2022)
- [Reliability-Adaptive Consistency Regularization for Weakly-Supervised Point Cloud Segmentation](https://mlsanity.com/paper/2303.05164/) (2023)
- [A Chebyshev Confidence Guided Source-Free Domain Adaptation Framework for Medical Image Segmentation](https://mlsanity.com/paper/2310.18087/) (2023)
- [Domain-Guided Weight Modulation for Semi-Supervised Domain Generalization](https://mlsanity.com/paper/2409.03509/) (2024)
- [Anticipating the Unseen Discrepancy for Vision and Language Navigation](https://mlsanity.com/paper/2209.04725/) (2022)
- [Distilling from Similar Tasks for Transfer Learning on a Budget](https://mlsanity.com/paper/2304.12314/) (2023)
- [DAC-MR: Data Augmentation Consistency Based Meta-Regularization for Meta-Learning](https://mlsanity.com/paper/2305.07892/) (2023)
- [Semi-supervised multimodal coreference resolution in image narrations](https://mlsanity.com/paper/2310.13619/) (2023)
- [CFTS-GAN: Continual Few-Shot Teacher Student for Generative Adversarial Networks](https://mlsanity.com/paper/2410.14749/) (2024)
- [Toward Robust Diagnosis: A Contour Attention Preserving Adversarial Defense for COVID-19 Detection](https://mlsanity.com/paper/2211.16806/) (2022)
- [Semi-Supervised Hybrid Spine Network for Segmentation of Spine MR Images](https://mlsanity.com/paper/2203.12151/) (2022)
- [MLN-net: A multi-source medical image segmentation method for clustered microcalcifications using multiple layer normalization](https://mlsanity.com/paper/2309.02742/) (2023)
- [Sequential Visual and Semantic Consistency for Semi-supervised Text Recognition](https://mlsanity.com/paper/2402.15806/) (2024)
- [Lagrange Duality and Compound Multi-Attention Transformer for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2409.07793/) (2024)
- [FedSemiDG: Domain Generalized Federated Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2501.07378/) (2025)
- [Few-Shot Learning in Video and 3D Object Detection: A Survey](https://mlsanity.com/paper/2507.17079/) (2025)
- [TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Data](https://mlsanity.com/paper/2111.04798/) (2021)
- [Augmentation-induced Consistency Regularization for Classification](https://mlsanity.com/paper/2205.12461/) (2022)
- [On the Regularization of Learnable Embeddings for Time Series Forecasting](https://mlsanity.com/paper/2410.14630/) (2024)
- [Multiview Self-Representation Learning across Heterogeneous Views](https://mlsanity.com/paper/2602.04328/) (2026)
- [A new semi-supervised inductive transfer learning framework: Co-Transfer](https://mlsanity.com/paper/2108.07930/) (2021)
- [Accessible Instruction-Following Agent](https://mlsanity.com/paper/2305.06358/) (2023)

## Abstract (arXiv)

While recent studies on semi-supervised learning have shown remarkable progress in leveraging both labeled and unlabeled data, most of them presume a basic setting of the model is randomly initialized. In this work, we consider semi-supervised learning and transfer learning jointly, leading to a more practical and competitive paradigm that can utilize both powerful pre-trained models from source domain as well as labeled/unlabeled data in the target domain. To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization that consists of two complementary components: Adaptive Knowledge Consistency (AKC) on the examples between the source and target model, and Adaptive Representation Consistency (ARC) on the target model between labeled and unlabeled examples. Examples involved in the consistency regularization are adaptively selected according to their potential contributions to the target task. We conduct extensive experiments on popular benchmarks including CIFAR-10, CUB-200, and MURA, by fine-tuning the ImageNet pre-trained ResNet-50 model. Results show that our proposed adaptive consistency regularization outperforms state-of-the-art semi-supervised learning techniques such as Pseudo Label, Mean Teacher, and FixMatch. Moreover, our algorithm is orthogonal to existing methods and thus able to gain additional improvements on top of MixMatch and FixMatch. Our code is available at https://github.com/SHI-Labs/Semi-Supervised-Transfer-Learning.