mlsanityData to 14 Oct 2026

Token Transformer: Can class token help window-based transformer build better long-range interactions?

arXiv 2211.06083 ↗First posted 11 Nov 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Compared with the vanilla transformer, the window-based transformer offers a better trade-off between accuracy and efficiency. Although the window-based transformer has made great progress, its long-range modeling capabilities are limited due to the size of the local window and the window connection scheme. To address this problem, we propose a novel Token Transformer (TT). The core mechanism of TT is the addition of a Class (CLS) token for summarizing window information in each local window. We refer to this type of token interaction as CLS Attention. These CLS tokens will interact spatially with the tokens in each window to enable long-range modeling. In order to preserve the hierarchical design of the window-based transformer, we designed Feature Inheritance Module (FIM) in each phase of TT to deliver the local window information from the previous phase to the CLS token in the next phase. In addition, we have designed a Spatial-Channel Feedforward Network (SCFFN) in TT, which can mix CLS tokens and embedded tokens on the spatial domain and channel domain without additional parameters. Extensive experiments have shown that our TT achieves competitive results with low parameters in image classification and downstream tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 39 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

38 of 39 references.

References · 38

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2017Mask R-CNNcited by 7,277
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
26 more
  1. 2021Going deeper with Image Transformerscited by 764
  2. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  3. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  4. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  5. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  6. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  7. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  8. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  9. 2022Vision Transformer with Deformable Attentioncited by 295
  10. 2017Dilated Residual Networkscited by 294
  11. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  12. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  13. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  14. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  15. 2022DaViT: Dual Attention Vision Transformerscited by 175
  16. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  17. 2022QuadTree Attention for Vision Transformerscited by 129
  18. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 127
  19. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  20. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  21. 2022Dilated Neighborhood Attention Transformercited by 64
  22. 2021Learned Queries for Efficient Local Attentioncited by 27
  23. 2022SepViT: Separable Vision Transformercited by 16
  24. 2021Local-to-Global Self-Attention in Vision Transformerscited by 13
  25. 2022Aggregating Global Features into Local Vision Transformercited by 10
  26. 2022Convolutional Embedding Makes Hierarchical Vision Transformer Strongercited by 10

Built on by

Not cited yet.