mlsanityData to 7 Oct 2026

Transformer in Transformer

arXiv 2103.00112 ↗First posted 27 Feb 2021cs.CV cs.AIcited by 632

Not on the map. Why?

Abstract (arXiv)

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16×16) as "visual sentences" and present to further divide them into smaller patches (e.g., 4×4) as "visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.

Key facts. Cited by 632 papers in the map (data through 7 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 27 Feb 2021.

Builds on

41 of 56 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2016Layer Normalizationcited by 5,579
29 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  13. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  14. 2017Random Erasing Data Augmentationcited by 1,235
  15. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  16. 2020Designing Network Design Spacescited by 954
  17. 2020Pre-Trained Image Processing Transformercited by 891
  18. 2019Are Sixteen Heads Really Better than One?cited by 803
  19. 2020A Survey on Visual Transformercited by 653
  20. 2013Network In Networkcited by 600
  21. 2019GhostNet: More Features from Cheap Operationscited by 410
  22. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  23. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  24. 2020End-to-end Lane Shape Prediction with Transformerscited by 111
  25. 2020Toward Transformer-Based Object Detectioncited by 84
  26. 2020Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetscited by 49
  27. 2020Grafit: Learning fine-grained image representations with coarse labelscited by 43
  28. 2019ReNAS:Relativistic Evaluation of Neural Architecture Searchcited by 35
  29. 2021Segmenting Transparent Object in the Wild with Transformercited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 124; 2022: 173; 2023: 161; 2024: 86; 2025: 66; 2026: 22.

632 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  5. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  8. 2021Going deeper with Image Transformerscited by 759
  9. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  10. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  11. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  12. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
38 more
  1. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  2. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  3. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  4. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  5. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  6. 2022DeiT III: Revenge of the ViTcited by 417
  7. 2021A Survey of Transformerscited by 399
  8. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  9. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  10. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  11. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  12. 2022Vision Transformer with Deformable Attentioncited by 293
  13. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  14. 2021Post-Training Quantization for Vision Transformercited by 274
  15. 2022Transformers in Medical Imaging: A Surveycited by 245
  16. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  17. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  18. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  19. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  20. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  21. 2022Visual Attention Networkcited by 210
  22. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  23. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  24. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  25. 2022DaViT: Dual Attention Vision Transformerscited by 174
  26. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  27. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  28. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  29. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  30. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  31. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  32. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  33. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 146
  34. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  35. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  36. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  37. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  38. 2021Towards Robust Vision Transformercited by 137

50 most cited of 632 All 632 (JSON)