mlsanityData to 14 Oct 2026

Discrete Representations Strengthen Vision Transformer Robustness

arXiv 2111.10493 ↗First posted 20 Nov 2021cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs trained on ImageNet are overly reliant on local textures and fail to make adequate use of shape information. ViTs thus have difficulties generalizing to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 20 Nov 2021.

Builds on

44 of 51 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
32 more
  1. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  11. 2019Natural Adversarial Examplescited by 1,408
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  13. 2020Quantifying Attention Flow in Transformerscited by 775
  14. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  15. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  16. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  17. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  18. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  19. 2021Intriguing Properties of Vision Transformerscited by 446
  20. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  21. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  22. 2020Are we done with ImageNet?cited by 333
  23. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  24. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  25. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 236
  26. 2021Vision Transformers are Robust Learnerscited by 207
  27. 2020Robust and Generalizable Visual Representation Learning via Random Convolutionscited by 142
  28. 2021Towards Robust Vision Transformercited by 137
  29. 2020Robust Learning Through Cross-Task Consistencycited by 105
  30. 2020Generative Interventions for Causal Learningcited by 45
  31. 2018DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priorscited by 20
  32. 2020Learning Visual Representations for Transfer Learning by Suppressing Texturecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 13; 2024: 8; 2025: 4; 2026: 3.

35 papers.

Sort:

By area:

  1. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  2. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 436
  3. 2022Test-Time Training with Masked Autoencoderscited by 212
  4. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  5. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 60
  6. 2023Online Clustered Codebookcited by 55
  7. 2021Pyramid Adversarial Training Improves ViT Performancecited by 39
  8. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  9. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  10. 2022Causal Transportability for Visual Recognitioncited by 23
  11. 2022Enhance the Visual Representation via Discrete Adversarial Trainingcited by 23
  12. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
23 more
  1. 2023Convolutional Visual Prompt for Robust Visual Perceptioncited by 17
  2. 2024GDA: Generalized Diffusion for Robust Test-time Adaptationcited by 14
  3. 2024Analyzing The Language of Visual Tokenscited by 12
  4. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  5. 2023Interpreting and Controlling Vision Foundation Models via Text Explanationscited by 11
  6. 2024[MASK] is All You Needcited by 11
  7. 2023Prototype-based Dataset Comparisoncited by 9
  8. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  9. 2025MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantizationcited by 9
  10. 2023Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformerscited by 7
  11. 2024Learning to Play Atari in a World of Tokenscited by 6
  12. 2022Deeper Insights into the Robustness of ViTs towards Common Corruptionscited by 5
  13. 2023BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervisioncited by 5
  14. 2024Vision Transformers with Natural Language Semanticscited by 5
  15. 2022Understanding Adversarial Robustness of Vision Transformers via Cauchy Problemcited by 4
  16. 2025Moment Quantization for Video Temporal Groundingcited by 4
  17. 2023RBFormer: Improve Adversarial Robustness of Transformer by Robust Biascited by 3
  18. 2025ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformerscited by 1
  19. 2026CONSTANT: Towards High-Quality One-Shot Handwriting Generation with Patch Contrastive Enhancement and Style-Aware Quantizationcited by 1
  20. 2024Towards Multi-Task Multi-Modal Models: A Video Generative Perspectivecited by 0
  21. 2025EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Visioncited by 0
  22. 2026ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margincited by 0
  23. 2026EEGDancer: Dynamic Emotion Latent Space Masked Modeling with Reinforcement Learning for EEG Continuous Emotion Predictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ViT-1.58b: Mobile Vision Transformers in the 1-bit Era
  2. 2026ViT-5: Vision Transformers for The Mid-2020s
  3. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformer
  4. 2024TPC-ViT: Token Propagation Controller for Efficient Vision Transformer
  5. 2026Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
  6. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformer
  7. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  8. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantization