mlsanityData to 7 Oct 2026

Aggregating Global Features into Local Vision Transformer

arXiv 2201.12903 ↗First posted 30 Jan 2022cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Local Transformer-based classification models have recently achieved promising results with relatively low computational costs. However, the effect of aggregating spatial global information of local Transformer-based architecture is not clear. This work investigates the outcome of applying a global attention-based module named multi-resolution overlapped attention (MOA) in the local window-based transformer after each stage. The proposed MOA employs slightly larger and overlapped patches in the key to enable neighborhood pixel information transmission, which leads to significant performance gain. In addition, we thoroughly investigate the effect of the dimension of essential architecture components through extensive experiments and discover an optimum architecture design. Extensive experimental results CIFAR-10, CIFAR-100, and ImageNet-1K datasets demonstrate that the proposed approach outperforms previous vision Transformers with a comparatively fewer number of parameters.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 30 Jan 2022.

Builds on

45 of 51 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2017Squeeze-and-Excitation Networkscited by 4,763
33 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2016Wide Residual Networkscited by 3,057
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  13. 2021Transformer in Transformercited by 632
  14. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  15. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  16. 2017Relation Networks for Object Detectioncited by 305
  17. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  18. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  19. 2019Local Relation Networks for Image Recognitioncited by 210
  20. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  21. 2018BAM: Bottleneck Attention Modulecited by 171
  22. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  23. 2021Colonoscopy Polyp Detection and Classification: Dataset Creation and Comparative Evaluationscited by 38
  24. 2021SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular imagescited by 33
  25. 2021Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentationcited by 33
  26. 2021Audio-Visual Transformer Based Crowd Countingcited by 12
  27. 2021Miti-DETR: Object Detection based on Transformers with Mitigatory Self-Attention Convergencecited by 12
  28. 2020Efficient Golf Ball Detection and Tracking Based on Convolutional Neural Networks and Kalman Filtercited by 8
  29. 2021A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wildcited by 8
  30. 2021A Domain Gap Aware Generative Adversarial Network for Multi-domain Image Translationcited by 8
  31. 2021Semantic Clustering based Deduction Learning for Image Recognition and Classificationcited by 8
  32. 2021A Discriminative Channel Diversification Network for Image Classificationcited by 4
  33. 2021Towards More Effective PRM-based Crowd Counting via A Multi-resolution Fusion and Attention Networkcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 3; 2024: 2.

10 papers.

  1. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  2. 2023HAT: Hybrid Attention Transformer for Image Restorationcited by 64
  3. 2022Accumulated Trivial Attention Matters in Vision Transformers on Small Datasetscited by 15
  4. 2024Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasetscited by 14
  5. 2022HaloAE: An HaloNet based Local Transformer Auto-Encoder for Anomaly Detection and Localizationcited by 5
  6. 2023PSLT: A Light-weight Vision Transformer with Ladder Self-Attention and Progressive Shiftcited by 5
  7. 2023Factorization Vision Transformer: Modeling Long Range Dependency with Local Window Costcited by 5
  8. 2024Guidance Disentanglement Network for Optics-Guided Thermal UAV Image Super-Resolutioncited by 2
  9. 2022Explicitly Increasing Input Information Density for Vision Transformers on Small Datasetscited by 1
  10. 2022Token Transformer: Can class token help window-based transformer build better long-range interactions?cited by 0