PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture
Not on the map. Why?
Abstract (arXiv)
Transformer networks have achieved great progress for computer vision tasks. Transformer-in-Transformer (TNT) architecture utilizes inner transformer and outer transformer to extract both local and global representations. In this work, we present new TNT baselines by introducing two advanced designs: 1) pyramid architecture, and 2) convolutional stem. The new "PyramidTNT" significantly improves the original TNT by establishing hierarchical representations. PyramidTNT achieves better performances than the previous state-of-the-art vision transformers such as Swin Transformer. We hope this new baseline will be helpful to the further research and application of vision transformer. Code will be available at https://github.com/huawei-noah/CV-Backbones/tree/master/tnt_pytorch.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 38 of its 41 references that are in the map. First posted on arXiv 4 Jan 2022.
Builds on
38 of 41 references.
References · 38
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2017Mask R-CNNcited by 7,259
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
26 more
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
- 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
- 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
- 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
- 2017Random Erasing Data Augmentationcited by 1,235
- 2018Self-Attention with Relative Position Representationscited by 1,112
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
- 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
- 2017Focal Loss for Dense Object Detectioncited by 749
- 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
- 2020A Survey on Visual Transformercited by 653
- 2021Transformer in Transformercited by 632
- 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2019GhostNet: More Features from Cheap Operationscited by 410
- 2021Early Convolutions Help Transformers See Bettercited by 395
- 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
- 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
- 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
- 2021S2-MLP: Spatial-Shift MLP Architecture for Visioncited by 83
- 2021Hire-MLP: Vision MLP via Hierarchical Rearrangementcited by 66
- 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61
- 2021Augmented Shortcuts for Vision Transformerscited by 19
- 2021Greedy Network Enlargingcited by 2
Built on by
Citing papers per year: 2022: 2; 2023: 1.
3 papers.