Towards Understanding the Effect of Pretraining Label Granularity
Not on the map. Why?
Abstract (arXiv)
In this paper, we study how the granularity of pretraining labels affects the generalization of deep neural networks in image classification tasks. We focus on the "fine-to-coarse" transfer learning setting, where the pretraining label space is more fine-grained than that of the target problem. Empirically, we show that pretraining on the leaf labels of ImageNet21k produces better transfer results on ImageNet1k than pretraining on other coarser granularity levels, which supports the common practice used in the community. Theoretically, we explain the benefit of fine-grained pretraining by proving that, for a data distribution satisfying certain hierarchy conditions, 1) coarse-grained pretraining only allows a neural network to learn the "common" or "easy-to-learn" features well, while 2) fine-grained pretraining helps the network learn the "rarer" or "fine-grained" features in addition to the common ones, thus improving its accuracy on hard downstream test samples in which common features are missing or weak in strength. Furthermore, we perform comprehensive experiments using the label hierarchies of iNaturalist 2021 and observe that the following conditions, in addition to proper choice of label granularity, enable the transfer to work well in practice: 1) the pretraining dataset needs to have a meaningful label hierarchy, and 2) the pretraining and target label functions need to align well.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 29 Mar 2023.
Builds on
41 of 62 references.
References · 41
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2014Going Deeper with Convolutionscited by 4,231
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
29 more
- 2021ImageNet-21K Pretraining for the Massescited by 607
- 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
- 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
- 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
- 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
- 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
- 2019A Survey of the Recent Architectures of Deep Convolutional Neural Networkscited by 211
- 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
- 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2018Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learningcited by 174
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
- 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
- 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
- 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
- 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
- 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
- 2017Do Convolutional Neural Networks Learn Class Hierarchy?cited by 61
- 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
- 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
- 2021Compressive Visual Representationscited by 41
- 2017B-CNN: Branch Convolutional Neural Network for Hierarchical Classificationcited by 37
- 2018Domain Adaptive Transfer Learning with Specialist Modelscited by 31
- 2019Understanding the Impact of Label Granularity on CNN-based Image Classificationcited by 13
- 2022Cluster & Tune: Boost Cold Start Performance in Text Classificationcited by 13
- 2022Reliable Decision from Multiple Subtasks through Threshold Optimization: Content Moderation in the Wildcited by 7
- 2020Hierarchical Class-Based Curriculum Losscited by 5
- 2021United We Learn Better: Harvesting Learning Improvements From Class Hierarchies Across Taskscited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.
3 papers.