mlsanityData to 7 Oct 2026

Dual PatchNorm

arXiv 2302.01327 ↗First posted 2 Feb 2023cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

26 of 30 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Layer Normalizationcited by 5,579
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2021Scaling Vision Transformerscited by 993
  12. 2020On Layer Normalization in the Transformer Architecturecited by 906
14 more
  1. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  2. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  3. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  4. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  5. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  6. 2021Early Convolutions Help Transformers See Bettercited by 395
  7. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  8. 2019Understanding and Improving Layer Normalizationcited by 213
  9. 2022FlexiViT: One Model for All Patch Sizescited by 151
  10. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  11. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  12. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  13. 2022Foundation Transformerscited by 19
  14. 2021Improved Robustness of Vision Transformer via PreLayerNorm in Patch Embeddingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 3.

9 papers.

  1. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 144
  2. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  3. 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
  4. 2024Learning from Offline Foundation Features with Tensor Augmentationscited by 2
  5. 2024Customize Your Own Paired Data via Few-shot Waycited by 0
  6. 2024TrAct: Making First-layer Pre-Activations Trainablecited by 0
  7. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0
  8. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  9. 2026Mind2Cloud: EEG-to-Point Cloud Generation with Two-Granularity Diffusion Decodingcited by 0