mlsanityData to 7 Oct 2026

SWAT: Spatial Structure Within and Among Tokens

arXiv 2111.13677 ↗First posted 26 Nov 2021cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Modeling visual data as tokens (i.e., image patches) using attention mechanisms, feed-forward networks or convolutions has been highly effective in recent years. Such methods usually have a common pipeline: a tokenization method, followed by a set of layers/blocks for information mixing, both within and among tokens. When image patches are converted into tokens, they are often flattened, discarding the spatial structure within each patch. As a result, any processing that follows (eg: multi-head self-attention) may fail to recover and/or benefit from such information. In this paper, we argue that models can have significant gains when spatial structure is preserved during tokenization, and is explicitly used during the mixing stage. We propose two key contributions: (1) Structure-aware Tokenization and, (2) Structure-aware Mixing, both of which can be combined with existing models with minimal effort. We introduce a family of models (SWAT), showing improvements over the likes of DeiT, MLP-Mixer and Swin Transformer, across multiple benchmarks including ImageNet classification and ADE20K segmentation. Our code is available at https://github.com/kkahatapitiya/SWAT.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 84 of its 111 references that are in the map. First posted on arXiv 26 Nov 2021.

Builds on

84 of 111 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  11. 2021Vision Transformers for Dense Predictioncited by 1,789
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  15. 2020Rethinking Attention with Performerscited by 1,584
  16. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  17. 2020Point Transformercited by 1,462
  18. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  19. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  20. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  21. 2021Video Swin Transformercited by 1,090
  22. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  23. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  24. 2016Deep Networks with Stochastic Depthcited by 973
  25. 2020Designing Network Design Spacescited by 954
  26. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  27. 2021Multiscale Vision Transformerscited by 896
  28. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  29. 2021Going deeper with Image Transformerscited by 759
  30. 2020PCT: Point cloud transformercited by 730
  31. 2016Language Modeling with Gated Convolutional Networkscited by 688
  32. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  33. 2021Transformer in Transformercited by 632
  34. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  35. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  36. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  37. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  38. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

4 papers.

  1. 2021MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detectioncited by 42
  2. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  3. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13
  4. 2026CAViT -- Channel-Aware Vision Transformer for Dynamic Feature Fusioncited by 0