# Efficient Conditional Pre-training for Transfer Learning

arXiv: https://arxiv.org/abs/2011.10231 · HTML: https://mlsanity.com/paper/2011.10231/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 11 of 11)

- [Self-Supervised Pre-Training for Transformer-Based Person Re-Identification](https://mlsanity.com/paper/2111.12084/) (2021)
- [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021)
- [Improving Contrastive Learning on Imbalanced Seed Data via Open-World Sampling](https://mlsanity.com/paper/2111.01004/) (2021)
- [Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain](https://mlsanity.com/paper/2207.03860/) (2022)
- [WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models](https://mlsanity.com/paper/2406.17503/) (2024)
- [Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks](https://mlsanity.com/paper/2311.01038/) (2023)
- [Transferring Core Knowledge via Learngenes](https://mlsanity.com/paper/2401.08139/) (2024)
- [Manipulating Transfer Learning for Property Inference](https://mlsanity.com/paper/2303.11643/) (2023)
- [Pretraining a Neural Operator in Lower Dimensions](https://mlsanity.com/paper/2407.17616/) (2024)
- [Scalable Neural Data Server: A Data Recommender for Transfer Learning](https://mlsanity.com/paper/2206.09386/) (2022)
- [MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.06096/) (2025)

## Abstract (arXiv)

Almost all the state-of-the-art neural networks for computer vision tasks are trained by (1) pre-training on a large-scale dataset and (2) finetuning on the target dataset. This strategy helps reduce dependence on the target dataset and improves convergence rate and generalization on the target task. Although pre-training on large-scale datasets is very useful, its foremost disadvantage is high training cost. To address this, we propose efficient filtering methods to select relevant subsets from the pre-training dataset. Additionally, we discover that lowering image resolutions in the pre-training step offers a great trade-off between cost and performance. We validate our techniques by pre-training on ImageNet in both the unsupervised and supervised settings and finetuning on a diverse collection of target datasets and tasks. Our proposed methods drastically reduce pre-training cost and provide strong performance boosts. Finally, we improve standard ImageNet pre-training by 1-3% by tuning available models on our subsets and pre-training on a dataset filtered from a larger scale dataset.