# Rethinking ImageNet Pre-training

arXiv: https://arxiv.org/abs/1811.08883 · HTML: https://mlsanity.com/paper/1811.08883/

## Key facts

- Cited by 377 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2018.

## Built on by (most cited first, 30 of 377)

- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [Efficient Visual Pretraining with Contrastive Detection](https://mlsanity.com/paper/2103.10957/) (2021)
- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [OmniVL:One Foundation Model for Image-Language and Video-Language Tasks](https://mlsanity.com/paper/2209.07526/) (2022)
- [An Empirical Investigation of the Role of Pre-training in Lifelong Learning](https://mlsanity.com/paper/2112.09153/) (2021)
- [Benchmarking Detection Transfer Learning with Vision Transformers](https://mlsanity.com/paper/2111.11429/) (2021)
- [AP-10K: A Benchmark for Animal Pose Estimation in the Wild](https://mlsanity.com/paper/2108.12617/) (2021)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions](https://mlsanity.com/paper/2206.08171/) (2022)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Instance Localization for Self-supervised Detection Pretraining](https://mlsanity.com/paper/2102.08318/) (2021)
- [MVP: Multimodality-guided Visual Pre-training](https://mlsanity.com/paper/2203.05175/) (2022)
- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [Self-Supervised Learning of Object Parts for Semantic Segmentation](https://mlsanity.com/paper/2204.13101/) (2022)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [Multi-Task Self-Training for Learning General Representations](https://mlsanity.com/paper/2108.11353/) (2021)

## Abstract (arXiv)

We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust; our results hold even when: (i) using only 10% of the training data, (ii) for deeper and wider models, and (iii) for multiple tasks and metrics. Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy. To push the envelope we demonstrate 50.9 AP on COCO object detection without using any external data---a result on par with the top COCO 2017 competition results that used ImageNet pre-training. These observations challenge the conventional wisdom of ImageNet pre-training for dependent tasks and we expect these discoveries will encourage people to rethink the current de facto paradigm of `pre-training and fine-tuning' in computer vision.