mlsanityData to 14 Oct 2026

MixFormer: Mixing Features across Windows and Dimensions

arXiv 2204.02557 ↗First posted 6 Apr 2022cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

While local-window self-attention performs notably in vision tasks, it suffers from limited receptive field and weak modeling capability issues. This is mainly because it performs self-attention within non-overlapped windows and shares weights on the channel dimension. We propose MixFormer to find a solution. First, we combine local-window self-attention with depth-wise convolution in a parallel design, modeling cross-window connections to enlarge the receptive fields. Second, we propose bi-directional interactions across branches to provide complementary clues in the channel and spatial dimensions. These two designs are integrated to achieve efficient feature mixing among windows and dimensions. Our MixFormer provides competitive results on image classification with EfficientNet and shows better results than RegNet and Swin Transformer. Performance in downstream tasks outperforms its alternatives by significant margins with less computational costs in 5 dense prediction tasks on MS COCO, ADE20k, and LVIS. Code is available at urlhttps://github.com/PaddlePaddle/PaddleClas.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 63 of its 68 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

63 of 68 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2017Mask R-CNNcited by 7,277
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  9. 2016Layer Normalizationcited by 5,669
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
38 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,789
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  8. 2017Non-local Neural Networkscited by 2,293
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  10. 2019Searching for MobileNetV3cited by 1,994
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  12. 2017Decoupled Weight Decay Regularizationcited by 1,831
  13. 2017Deformable Convolutional Networkscited by 1,488
  14. 2015Spatial Transformer Networkscited by 1,429
  15. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  16. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  17. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  18. 2017Random Erasing Data Augmentationcited by 1,239
  19. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  20. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  21. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  22. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  23. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  24. 2016Deep Networks with Stochastic Depthcited by 979
  25. 2020Designing Network Design Spacescited by 960
  26. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  27. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  28. 2021Transformer in Transformercited by 633
  29. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  30. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  31. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  32. 2019Selective Kernel Networkscited by 403
  33. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  34. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  35. 2020Exploring Self-attention for Image Recognitioncited by 313
  36. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  37. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  38. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 14; 2024: 13; 2025: 5; 2026: 2.

43 papers.

Sort:

By area:

  1. 2023Dual Aggregation Transformer for Image Super-Resolutioncited by 124
  2. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 113
  3. 2022Changer: Feature Interaction is What You Need for Change Detectioncited by 79
  4. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  5. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 66
  6. 2024HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolutioncited by 42
  7. 2023Xformer: Hybrid X-Shaped Transformer for Image Denoisingcited by 37
  8. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  9. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 29
  10. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  11. 2023Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Countingcited by 25
  12. 2022Residual Degradation Learning Unfolding Framework with Mixing Priors across Spectral and Spatial for Compressive Spectral Imagingcited by 22
31 more
  1. 2023TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognitioncited by 21
  2. 2025CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolutioncited by 16
  3. 2023Dual Cross-Attention for Medical Image Segmentationcited by 15
  4. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  5. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
  6. 2024Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasetscited by 14
  7. 2025iFormer: Integrating ConvNet and Transformer for Mobile Applicationcited by 9
  8. 2022AIM 2022 Challenge on Instagram Filter Removal: Methods and Resultscited by 6
  9. 2023ConTrack: Contextual Transformer for Device Tracking in X-raycited by 4
  10. 2022IncepFormer: Efficient Inception Transformer with Pyramid Pooling for Semantic Segmentationcited by 3
  11. 2023Vision Backbone Enhancement via Multi-Stage Cross-Scale Attentioncited by 3
  12. 2024ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connectionscited by 3
  13. 2024FViT: A Focal Vision Transformer with Gabor Filtercited by 3
  14. 2024SRC-Net: Bi-Temporal Spatial Relationship Concerned Network for Change Detectioncited by 3
  15. 2022ITSRN++: Stronger and Better Implicit Transformer Network for Continuous Screen Content Image Super-Resolutioncited by 2
  16. 2023Compressing Vision Transformers for Low-Resource Visual Learningcited by 2
  17. 2023Contrastive Learning-Based Spectral Knowledge Distillation for Multi-Modality and Missing Modality Scenarios in Semantic Segmentationcited by 2
  18. 2025ViT-Split: Unleashing the Power of Vision Foundation Models via Efficient Splitting Headscited by 2
  19. 2022Dunhuang murals contour generation network based on convolution and self-attention fusioncited by 1
  20. 2023EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attentioncited by 1
  21. 2024Attention-Enhanced Hybrid Feature Aggregation Network for 3D Brain Tumor Segmentationcited by 1
  22. 2024A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusioncited by 1
  23. 2024AFIDAF: Alternating Fourier and Image Domain Adaptive Filters as an Efficient Alternative to Attention in ViTscited by 1
  24. 2024Joint multi-dimensional dynamic attention and transformer for general image restorationcited by 1
  25. 2025SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averagingcited by 1
  26. 2025A2Mamba: Attention-augmented State Space Models for Visual Recognitioncited by 1
  27. 2023Vision Transformers for Mobile Applications: A Short Surveycited by 0
  28. 2024ParFormer: A Vision Transformer with Parallel Mixer and Sparse Channel Attention Patch Embeddingcited by 0
  29. 2024BA-Net: Bridge Attention in Deep Neural Networkscited by 0
  30. 2026Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformercited by 0
  31. 2026Advancing Vision Transformer with Enhanced Spatial Priorscited by 0