mlsanityData to 7 Oct 2026

Deep Transformers Thirst for Comprehensive-Frequency Data

arXiv 2203.07116 ↗First posted 14 Mar 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Current researches indicate that inductive bias (IB) can improve Vision Transformer (ViT) performance. However, they introduce a pyramid structure concurrently to counteract the incremental FLOPs and parameters caused by introducing IB. This structure destroys the unification of computer vision and natural language processing (NLP) and complicates the model. We study an NLP model called LSRA, which introduces IB with a pyramid-free structure. We analyze why it outperforms ViT, discovering that introducing IB increases the share of high-frequency data in each layer, giving "attention" to more information. As a result, the heads notice more diverse information, showing better performance. To further explore the potential of transformers, we propose EIT, which Efficiently introduces IB to ViT with a novel decreasing convolutional structure under a pyramid-free structure. EIT achieves competitive performance with the state-of-the-art (SOTA) methods on ImageNet-1K and achieves SOTA performance over the same scale models which have the pyramid-free structure.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

30 of 39 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
18 more
  1. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  2. 2017Random Erasing Data Augmentationcited by 1,235
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2020Designing Network Design Spacescited by 954
  5. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  6. 2020Pre-Trained Image Processing Transformercited by 891
  7. 2018Image Transformercited by 770
  8. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  9. 2021Transformer in Transformercited by 632
  10. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  11. 2021Early Convolutions Help Transformers See Bettercited by 395
  12. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  13. 2022How Do Vision Transformers Work?cited by 305
  14. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  15. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  16. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  17. 2020Embedded Encoder-Decoder in Convolutional Networks Towards Explainable AIcited by 23
  18. 2021Evolving Attention with Residual Convolutionscited by 20

Built on by

Not cited yet.