mlsanityData to 7 Oct 2026

High-Performance Large-Scale Image Recognition Without Normalization

arXiv 2102.06171 ↗First posted 11 Feb 2021cs.CV cs.LG stat.MLcited by 359

Not on the map. Why?

Abstract (arXiv)

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). Builds on 71 of its 101 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

71 of 101 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  13. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  14. 2018Group Normalizationcited by 1,514
  15. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  16. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  17. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  18. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  19. 2016Deep Networks with Stochastic Depthcited by 973
  20. 2020Designing Network Design Spacescited by 954
  21. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  22. 2018Self-Attention Generative Adversarial Networkscited by 809
  23. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  24. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  25. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  26. 2017Large Batch Training of Convolutional Networkscited by 529
  27. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  28. 2020Meta Pseudo Labelscited by 372
  29. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  30. 2018How Does Batch Normalization Help Optimization?cited by 355
  31. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  32. 2016On the Expressive Power of Deep Neural Networkscited by 302
  33. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  34. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  35. 2015Highway Networkscited by 257
  36. 2019Fixing the train-test resolution discrepancycited by 209
  37. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  38. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 78; 2022: 94; 2023: 68; 2024: 59; 2025: 33; 2026: 27.

359 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2023Mastering Diverse Domains through World Modelscited by 1,193
  4. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2021Going deeper with Image Transformerscited by 759
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  12. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
38 more
  1. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  2. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  3. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  5. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  6. 2021Pay Attention to MLPscited by 367
  7. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  8. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  9. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  10. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  11. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  12. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  13. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  14. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  15. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  16. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  17. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  18. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  19. 2022DaViT: Dual Attention Vision Transformerscited by 174
  20. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  21. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  22. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  23. 2021Revisiting ResNets: Improved Training and Scaling Strategiescited by 158
  24. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  25. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  26. 2022Linearly Mapping from Image to Text Spacecited by 139
  27. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  28. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  29. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  30. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  31. 2021Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognitioncited by 103
  32. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  33. 2021CAT: Cross Attention in Vision Transformercited by 92
  34. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  35. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  36. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  37. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 83
  38. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82

50 most cited of 359 All 359 (JSON)