mlsanityData to 7 Oct 2026

TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers

arXiv 2210.07562 ↗First posted 14 Oct 2022cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

Mixup is a commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation and are especially burdensome for parameter-heavy transformer models. To this end, we propose TokenMixup, an efficient attention-guided token-level data augmentation method that aims to maximize the saliency of a mixed set of tokens. TokenMixup provides x15 faster saliency-aware data augmentation compared to gradient-based methods. Moreover, we introduce a variant of TokenMixup which mixes tokens within a single instance, thereby enabling multi-scale feature augmentation. Experiments show that our methods significantly improve the baseline models' performance on CIFAR and ImageNet-1K, while being more efficient than previous methods. We also reach state-of-the-art performance on CIFAR-100 among from-scratch transformer models. Code is available at https://github.com/mlvlab/TokenMixup.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

42 of 63 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2017Squeeze-and-Excitation Networkscited by 4,763
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2016Wide Residual Networkscited by 3,057
30 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  3. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2020Designing Network Design Spacescited by 954
  8. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  10. 2020Quantifying Attention Flow in Transformerscited by 759
  11. 2021Transformer in Transformercited by 632
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  13. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  14. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  15. 2017Residual Attention Network for Image Classificationcited by 385
  16. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  17. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  18. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  19. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 260
  20. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  21. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  22. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  23. 2020SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularizationcited by 149
  24. 2021Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversitycited by 127
  25. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  26. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  27. 2019Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Trainingcited by 95
  28. 2020SuperMix: Supervising the Mixing Data Augmentationcited by 58
  29. 2021PnP-DETR: Towards Efficient Visual Analysis with Transformerscited by 52
  30. 2022Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detectioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 10; 2025: 4; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024DiffuseMix: Label-Preserving Data Augmentation with Diffusion Modelscited by 78
  2. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  3. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  4. 2024Prompt Learning via Meta-Regularizationcited by 26
  5. 2023Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classificationcited by 24
  6. 2024A Survey on Mixup Augmentations and Beyondcited by 18
  7. 2024SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detectioncited by 17
  8. 2023Adversarial AutoMixupcited by 16
  9. 2023TFS-ViT: Token-Level Feature Stylization for Domain Generalizationcited by 10
  10. 2025Context-guided Responsible Data Augmentation with Diffusion Modelscited by 9
  11. 2023Inter-Instance Similarity Modeling for Contrastive Learningcited by 6
  12. 2024Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Modelscited by 5
14 more
  1. 2025SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixingcited by 5
  2. 2025DiSa: Directional Saliency-Aware Prompt Learning for Generalizable Vision-Language Modelscited by 4
  3. 2024SETA: Semantic-Aware Token Augmentation for Domain Generalizationcited by 3
  4. 2024SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipecited by 3
  5. 2025GLAD: Generalizable Tuning for Vision-Language Modelscited by 3
  6. 2023Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretabilitycited by 2
  7. 2023AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformercited by 1
  8. 2023Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Workcited by 0
  9. 2023Patch Is Not All You Needcited by 0
  10. 2024Random Token Fusion for Multi-View Medical Diagnosiscited by 0
  11. 2024Integrated Image-Text Based on Semi-supervised Learning for Small Sample Instance Segmentationcited by 0
  12. 2026CASP: Few-Shot Class-Incremental Learning with CLS Token Attention Steering Promptscited by 0
  13. 2026Harnessing Lightweight Transformer with Contextual Synergic Enhancement for Efficient 3D Medical Image Segmentationcited by 0
  14. 2026MedDiffuseMix: Preserving Diagnostic Evidence with Saliency-Aware Diffusion Medical Image Data Augmentationcited by 0