mlsanityData to 14 Oct 2026

Brain-Inspired Stepwise Patch Merging for Vision Transformers

arXiv 2409.06963 ↗First posted 11 Sep 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The hierarchical architecture has become a mainstream design paradigm for Vision Transformers (ViTs), with Patch Merging serving as the pivotal component that transforms a columnar architecture into a hierarchical one. Drawing inspiration from the brain's ability to integrate global and local information for comprehensive visual understanding, we propose Stepwise Patch Merging (SPM), which enhances the subsequent attention mechanism's ability to 'see' better. SPM consists of Multi-Scale Aggregation (MSA) and Guided Local Enhancement (GLE) striking a proper balance between long-range dependency modeling and local feature enhancement. Extensive experiments conducted on benchmark datasets, including ImageNet-1K, COCO, and ADE20K, demonstrate that SPM significantly improves the performance of various models, particularly in dense prediction tasks such as object detection and semantic segmentation. Meanwhile, experiments show that combining SPM with different backbones can further improve performance. The code has been released at https://github.com/Yonghao-Yu/StepwisePatchMerging.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

41 of 56 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2017Mask R-CNNcited by 7,277
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
29 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2014Going Deeper with Convolutionscited by 4,241
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  6. 2017Deformable Convolutional Networkscited by 1,488
  7. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  8. 2017Random Erasing Data Augmentationcited by 1,239
  9. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  10. 2022Token Merging: Your ViT But Fastercited by 1,021
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  13. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 670
  14. 2019Panoptic Feature Pyramid Networkscited by 645
  15. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
  16. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  17. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  18. 2019Selective Kernel Networkscited by 403
  19. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  20. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  21. 2022Neighborhood Attention Transformercited by 239
  22. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 101
  23. 2021Shunted Self-Attention via Multi-Scale Token Aggregationcited by 90
  24. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  25. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  26. 2021Less is More: Pay Less Attention in Vision Transformerscited by 53
  27. 2023Scale-Aware Modulation Meet Transformercited by 49
  28. 2021Vision Transformer with Progressive Samplingcited by 44
  29. 2021ConTNet: Why not use convolution and transformer at the same time?cited by 38

Built on by

Not cited yet.