mlsanityData to 7 Oct 2026

Improving Vision Transformers by Revisiting High-frequency Components

arXiv 2204.00993 ↗First posted 3 Apr 2022cs.CVcited by 75

Not on the map. Why?

Abstract (arXiv)

The transformer models have shown promising effectiveness in dealing with various vision tasks. However, compared with training Convolutional Neural Network (CNN) models, training Vision Transformer (ViT) models is more difficult and relies on the large-scale training set. To explain this observation we make a hypothesis that textitViT models are less effective in capturing the high-frequency components of images than CNN models, and verify it by a frequency analysis. Inspired by this finding, we first investigate the effects of existing techniques for improving ViT models from a new frequency perspective, and find that the success of some techniques (e.g., RandAugment) can be attributed to the better usage of the high-frequency components. Then, to compensate for this insufficient ability of ViT models, we propose HAT, which directly augments high-frequency components of images via adversarial training. We show that HAT can consistently boost the performance of various ViT models (e.g., +1.2% for ViT-B, +0.5% for Swin-B), and especially enhance the advanced model VOLO-D5 to 87.3% that only uses ImageNet-1K data, and the superiority can also be maintained on out-of-distribution data and transferred to downstream tasks. The code is available at: https://github.com/jiawangbai/HAT.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 76 of its 93 references that are in the map. First posted on arXiv 3 Apr 2022.

Builds on

76 of 93 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  15. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  16. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  17. 2019Natural Adversarial Examplescited by 1,401
  18. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  19. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  20. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  21. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  22. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  23. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  24. 2016Deep Networks with Stochastic Depthcited by 973
  25. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  26. 2020Designing Network Design Spacescited by 954
  27. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  28. 2020Pre-Trained Image Processing Transformercited by 891
  29. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  30. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  31. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  32. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  33. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  34. 2018Rethinking ImageNet Pre-trainingcited by 376
  35. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  36. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  37. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  38. 2020Are we done with ImageNet?cited by 329

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 24; 2025: 21; 2026: 6.

75 papers.

Sort:

By area:

  1. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  2. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  3. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  4. 2023Backdoor Defense via Adaptively Splitting Poisoned Datasetcited by 43
  5. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  6. 2023ALOFT: A Lightweight MLP-like Architecture with Dynamic Low-frequency Transform for Domain Generalizationcited by 28
  7. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  8. 2022Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentationcited by 19
  9. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  10. 2024Dual-Path Coupled Image Deraining Network via Spatial-Frequency Interactioncited by 16
  11. 2024MLP Can Be A Good Transformer Learnercited by 16
  12. 2023SPANet: Frequency-balancing Token Mixer using Spectral Pooling Aggregation Modulationcited by 14
38 more
  1. 2024All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Modelcited by 13
  2. 2022Exploring the Coordination of Frequency and Attention in Masked Image Modelingcited by 12
  3. 2023Improving Adversarial Robustness of Masked Autoencoders via Test-time Frequency-domain Promptingcited by 12
  4. 2024Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentationcited by 12
  5. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  6. 2024Exploring the Adversarial Robustness of CLIP for AI-generated Image Detectioncited by 11
  7. 2023Laplacian-Former: Overcoming the Limitations of Vision Transformers in Local Texture Detectioncited by 10
  8. 2025Frequency-Dynamic Attention Modulation for Dense Predictioncited by 10
  9. 2023FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognitioncited by 9
  10. 2024Generalization Boosted Adapter for Open-Vocabulary Segmentationcited by 9
  11. 2024Improving Representation of High-frequency Components for Medical Visual Foundation Modelscited by 8
  12. 2025Mamba-Sea: A Mamba-based Framework with Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentationcited by 7
  13. 2025Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspectivecited by 7
  14. 2023Efficiency 360: Efficient Vision Transformerscited by 6
  15. 2023Illumination Distillation Framework for Nighttime Person Re-Identification and A New Benchmarkcited by 6
  16. 2024Adaptive Parametric Activation: Unifying and Generalising Activation Functions Across Taskscited by 6
  17. 2023Tied-Augment: Controlling Representation Similarity Improves Data Augmentationcited by 5
  18. 2024Learning Generalizable Models via Disentangling Spurious and Enhancing Potential Correlationscited by 5
  19. 2024Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformerscited by 5
  20. 2022Similarity of Neural Architectures using Adversarial Attack Transferabilitycited by 4
  21. 2022Representation Separation for Semantic Segmentation with Vision Transformerscited by 4
  22. 2024Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approachcited by 4
  23. 2024START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformationcited by 4
  24. 2025Frequency-Compensated Network for Daily Arctic Sea Ice Concentration Predictioncited by 4
  25. 2023ScatterFormer: Locally-Invariant Scattering Transformer for Patient-Independent Multispectral Detection of Epileptiform Dischargescited by 3
  26. 2023Revisiting Vision Transformer from the View of Path Ensemblecited by 3
  27. 2024Navigating Beyond Dropout: An Intriguing Solution Towards Generalizable Image Super Resolutioncited by 3
  28. 2024SETA: Semantic-Aware Token Augmentation for Domain Generalizationcited by 3
  29. 2024Mitigating Low-Frequency Bias: Feature Recalibration and Frequency Attention Regularization for Adversarial Robustnesscited by 3
  30. 2025DG-DETR: Toward Domain Generalized Detection Transformercited by 3
  31. 2023Towards an Effective and Efficient Transformer for Rain-by-snow Weather Removalcited by 2
  32. 2023Multiscale Memory Comparator Transformer for Few-Shot Video Segmentationcited by 2
  33. 2024Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinementcited by 2
  34. 2025WaveFormer: A 3D Transformer with Wavelet-Driven Feature Representation for Efficient Medical Image Segmentationcited by 2
  35. 2026S2-FracMix: Label-Preserving Self-Saliency Mixup Augmentationcited by 2
  36. 2023Img2Vec: A Teacher of High Token-Diversity Helps Masked AutoEncoderscited by 1
  37. 2024Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognitioncited by 1
  38. 2024Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptationcited by 1

50 most cited of 75 All 75 (JSON)