mlsanityData to 14 Oct 2026

Co-training 2L Submodels for Visual Recognition

arXiv 2212.04884 ↗First posted 9 Dec 2022cs.CVcited by 8

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

We introduce submodel co-training, a regularization method related to co-training, self-distillation and stochastic depth. Given a neural network to be trained, for each sample we implicitly instantiate two altered networks, ``submodels'', with stochastic depth: we activate only a subset of the layers. Each network serves as a soft teacher to the other, by providing a loss that complements the regular loss provided by the one-hot label. Our approach, dubbed cosub, uses a single set of weights, and does not involve a pre-trained external model or temporal averaging. Experimentally, we show that submodel co-training is effective to train backbones for recognition tasks such as image classification and semantic segmentation. Our approach is compatible with multiple architectures, including RegNet, ViT, PiT, XCiT, Swin and ConvNext. Our training strategy improves their results in comparable settings. For instance, a ViT-B pretrained with cosub on ImageNet-21k obtains 87.4% top-1 acc. @448 on ImageNet-val.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 44 of its 65 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

44 of 65 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2022A ConvNet for the 2020scited by 3,579
32 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  8. 2014How transferable are features in deep neural networks?cited by 1,225
  9. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  10. 2016Deep Networks with Stochastic Depthcited by 979
  11. 2020Designing Network Design Spacescited by 960
  12. 2021Going deeper with Image Transformerscited by 764
  13. 2017Deep Mutual Learningcited by 628
  14. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  15. 2013Do Deep Nets Really Need to be Deep?cited by 474
  16. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  17. 2022DeiT III: Revenge of the ViTcited by 419
  18. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  19. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  20. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  21. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  22. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  23. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
  24. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  25. 2018Large scale distributed neural network training through online distillationcited by 146
  26. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  27. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  28. 2021Training Vision Transformers for Image Retrievalcited by 90
  29. 2018Collaborative Learning for Deep Neural Networkscited by 69
  30. 2020Transferring Inductive Biases through Knowledge Distillationcited by 41
  31. 2019Revisiting Knowledge Distillation via Label Smoothing Regularizationcited by 29
  32. 2020Circumventing Outliers of AutoAugment with Knowledge Distillationcited by 29

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4; 2025: 1.

8 papers.

  1. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51
  2. 2024CR-CTC: Consistency regularization on CTC for improved speech recognitioncited by 17
  3. 2023Morphing Tokens Draw Strong Masked Image Modelscited by 4
  4. 2023Learning with Unmasked Tokens Drives Stronger Vision Learnerscited by 3
  5. 2024MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentationcited by 2
  6. 2023Adaptive Depth Networks with Skippable Sub-Pathscited by 1
  7. 2024POA: Pre-training Once for Models of All Sizescited by 1
  8. 2025Improving Audio Event Recognition with Consistency Regularizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts
  2. 2025Co-Training with Active Contrastive Learning and Meta-Pseudo-Labeling on 2D Projections for Deep Semi-Supervised Learning
  3. 2021SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation
  4. 2026Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models
  5. 2026V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising
  6. 2021X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation
  7. 2026ACD-U: Asymmetric co-teaching with machine unlearning for robust learning with noisy labels
  8. 2024Hybrid diffusion models: combining supervised and generative pretraining for label-efficient fine-tuning of segmentation models