Dual Path Transformer with Partition Attention
Not on the map. Why?
Abstract (arXiv)
This paper introduces a novel attention mechanism, called dual attention, which is both efficient and effective. The dual attention mechanism consists of two parallel components: local attention generated by Convolutional Neural Networks (CNNs) and long-range attention generated by Vision Transformers (ViTs). To address the high computational complexity and memory footprint of vanilla Multi-Head Self-Attention (MHSA), we introduce a novel Multi-Head Partition-wise Attention (MHPA) mechanism. The partition-wise attention approach models both intra-partition and inter-partition attention simultaneously. Building on the dual attention block and partition-wise attention mechanism, we present a hierarchical vision backbone called DualFormer. We evaluate the effectiveness of our model on several computer vision tasks, including image classification on ImageNet, object detection on COCO, and semantic segmentation on Cityscapes. Specifically, the proposed DualFormer-XS achieves 81.5% top-1 accuracy on ImageNet, outperforming the recent state-of-the-art MPViT-XS by 0.6% top-1 accuracy with much higher throughput.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 55 of its 63 references that are in the map. First posted on arXiv 24 May 2023.
Builds on
55 of 63 references.
References · 55
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2017Mask R-CNNcited by 7,277
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2016Feature Pyramid Networks for Object Detectioncited by 5,036
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2017Squeeze-and-Excitation Networkscited by 4,789
38 more
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
- 2020Reformer: The Efficient Transformercited by 1,668
- 2020Rethinking Attention with Performerscited by 1,601
- 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
- 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
- 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
- 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
- 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
- 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
- 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
- 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
- 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
- 2021MetaFormer Is Actually What You Need for Visioncited by 524
- 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 495
- 2022MaxViT: Multi-Axis Vision Transformercited by 476
- 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
- 2021Early Convolutions Help Transformers See Bettercited by 396
- 2018Efficient Attention: Attention with Linear Complexitiescited by 368
- 2021XCiT: Cross-Covariance Image Transformerscited by 338
- 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
- 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
- 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
- 2022How Do Vision Transformers Work?cited by 307
- 2022Vision Transformer with Deformable Attentioncited by 295
- 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
- 2021Incorporating Convolution Designs into Visual Transformerscited by 248
- 2021Co-Scale Conv-Attentional Image Transformerscited by 224
- 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 184
- 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
- 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
- 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
- 2022Inception Transformercited by 99
- 2021Is Attention Better Than Matrix Decomposition?cited by 77
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.