Fully Attentional Networks with Self-emerging Token Labeling
Not on the map. Why?
Abstract (arXiv)
Recent studies indicate that Vision Transformers (ViTs) are robust against out-of-distribution scenarios. In particular, the Fully Attentional Network (FAN) - a family of ViT backbones, has achieved state-of-the-art robustness. In this paper, we revisit the FAN models and improve their pre-training with a self-emerging token labeling (STL) framework. Our method contains a two-stage training framework. Specifically, we first train a FAN token labeler (FAN-TL) to generate semantically meaningful patch token labels, followed by a FAN student model training stage that uses both the token labels and the original class label. With the proposed STL framework, our best model based on FAN-L-Hybrid (77.3M parameters) achieves 84.8% Top-1 accuracy and 42.1% mCE on ImageNet-1K and ImageNet-C, and sets a new state-of-the-art for ImageNet-A (46.1%) and ImageNet-R (56.6%) without using extra data, outperforming the original FAN counterpart by significant margins. The proposed framework also demonstrates significantly enhanced performance on downstream tasks such as semantic segmentation, with up to 1.7% improvement in robustness over the counterpart model. Code is available at https://github.com/NVlabs/STL.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 55 of its 66 references that are in the map. First posted on arXiv 8 Jan 2024.
Builds on
55 of 66 references.
References · 55
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2017Mask R-CNNcited by 7,277
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
38 more
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2022A ConvNet for the 2020scited by 3,579
- 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
- 2016Pyramid Scene Parsing Networkcited by 2,479
- 2015Learning Deep Features for Discriminative Localizationcited by 2,436
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
- 2019Natural Adversarial Examplescited by 1,408
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
- 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
- 2017Random Erasing Data Augmentationcited by 1,239
- 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
- 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
- 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
- 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
- 2021Intriguing Properties of Vision Transformerscited by 446
- 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
- 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
- 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
- 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
- 2021Vision Transformers are Robust Learnerscited by 207
- 2021Are Transformers More Robust Than CNNs?cited by 187
- 2016Wider or Deeper: Revisiting the ResNet Model for Visual Recognitioncited by 184
- 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
- 2022Understanding The Robustness in Vision Transformerscited by 144
- 2021Towards Robust Vision Transformercited by 137
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.