mlsanityData to 7 Oct 2026

ResNet strikes back: An improved training procedure in timm

arXiv 2110.00476 ↗First posted 1 Oct 2021cs.CV cs.LGcited by 365

Not on the map. Why?

Abstract (arXiv)

The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization & data-augmentation have increased the effectiveness of the training recipes. In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224x224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.

Key facts. Cited by 365 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

48 of 62 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
36 more
  1. 2020Supervised Contrastive Learningcited by 2,803
  2. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2017Random Erasing Data Augmentationcited by 1,235
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  12. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  13. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  14. 2016Deep Networks with Stochastic Depthcited by 973
  15. 2020Designing Network Design Spacescited by 954
  16. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  17. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  18. 2021Going deeper with Image Transformerscited by 759
  19. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  20. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  21. 2020ResNeSt: Split-Attention Networkscited by 445
  22. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  23. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  24. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  25. 2020Are we done with ImageNet?cited by 329
  26. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  27. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  28. 2019Fixing the train-test resolution discrepancycited by 209
  29. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  30. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  31. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  32. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 76
  33. 2021Fast and Accurate Model Scalingcited by 46
  34. 2020Grafit: Learning fine-grained image representations with coarse labelscited by 43
  35. 2020Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Networkcited by 24
  36. 2020AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weightscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 109; 2023: 103; 2024: 61; 2025: 48; 2026: 26.

365 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  4. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  5. 2022DeiT III: Revenge of the ViTcited by 417
  6. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  7. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  8. 2024Rotary Position Embedding for Vision Transformercited by 226
  9. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  10. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  11. 2022Knowledge Distillation from A Stronger Teachercited by 204
  12. 2022Patches Are All You Need?cited by 201
38 more
  1. 2022RTMDet: An Empirical Study of Designing Real-Time Object Detectorscited by 192
  2. 2022Focal Modulation Networkscited by 186
  3. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  4. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 175
  5. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  6. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  7. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  8. 2023SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligencecited by 163
  9. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  10. 2022MetaFormer Baselines for Visioncited by 131
  11. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 129
  12. 2022Retrieval Augmented Classification for Long-Tail Visual Recognitioncited by 106
  13. 2022Inception Transformercited by 99
  14. 2021The Efficiency Misnomercited by 98
  15. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  16. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  17. 2023Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modelingcited by 92
  18. 2022Three things everyone should know about Vision Transformerscited by 90
  19. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  20. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  21. 2023InceptionNeXt: When Inception Meets ConvNeXtcited by 84
  22. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  23. 2021ML-Decoder: Scalable and Versatile Classification Headcited by 78
  24. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  25. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 75
  26. 2022NAFSSR: Stereo Image Super-Resolution Using NAFNetcited by 75
  27. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  28. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  29. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 70
  30. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  31. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  32. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  33. 2024When Do We Not Need Larger Vision Models?cited by 65
  34. 2022A Light Recipe to Train Robust Vision Transformerscited by 64
  35. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  36. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 62
  37. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  38. 2023Mimetic Initialization of Self-Attention Layerscited by 52

50 most cited of 365 All 365 (JSON)