mlsanityData to 14 Oct 2026

Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders

arXiv 2310.20704 ↗First posted 31 Oct 2023cs.CV cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) have become ubiquitous in computer vision. Despite their success, ViTs lack inductive biases, which can make it difficult to train them with limited data. To address this challenge, prior studies suggest training ViTs with self-supervised learning (SSL) and fine-tuning sequentially. However, we observe that jointly optimizing ViTs for the primary task and a Self-Supervised Auxiliary Task (SSAT) is surprisingly beneficial when the amount of training data is limited. We explore the appropriate SSL tasks that can be optimized alongside the primary task, the training schemes for these tasks, and the data scale at which they can be most effective. Our findings reveal that SSAT is a powerful technique that enables ViTs to leverage the unique characteristics of both the self-supervised and primary tasks, achieving better performance than typical ViTs pre-training with SSL and fine-tuning sequentially. Our experiments, conducted on 10 datasets, demonstrate that SSAT significantly improves ViT performance while reducing carbon footprint. We also confirm the effectiveness of SSAT in the video domain for deepfake detection, showcasing its generalizability. Our code is available at https://github.com/dominickrei/Limited-data-vits.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 57 of its 67 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

57 of 67 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2021ViViT: A Video Vision Transformercited by 1,620
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  9. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,101
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  11. 2021Video Swin Transformercited by 1,091
  12. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  14. 2021Multiscale Vision Transformerscited by 897
  15. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  16. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  17. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  18. 2021Transformer in Transformercited by 633
  19. 2018Albumentations: fast and flexible image augmentationscited by 560
  20. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  21. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  22. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  23. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  24. 2022How Do Vision Transformers Work?cited by 307
  25. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  26. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  27. 2021Vision Transformer for Small-Size Datasetscited by 122
  28. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  29. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  30. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  31. 2021Combining EfficientNet and Vision Transformers for Video Deepfake Detectioncited by 82
  32. 2020Evolving Losses for Unsupervised Video Representation Learningcited by 74
  33. 2021Self-supervised Video Transformercited by 64
  34. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  35. 2022Self-Supervised Vision Transformers Learn Visual Concepts in Histopathologycited by 50
  36. 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
  37. 2021MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detectioncited by 42
  38. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 40

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 5.

17 papers.

Sort:

By area:

  1. 2024Exploring Self-Supervised Vision Transformers for Deepfake Detection: A Comparative Analysiscited by 15
  2. 2024An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Trainingcited by 4
  3. 2024Self-supervised Auxiliary Learning for Texture and Model-based Hybrid Robust and Fair Featuring in Face Analysiscited by 3
  4. 2024SSAD: Self-supervised Auxiliary Detection Framework for Panoramic X-ray based Dental Disease Diagnosiscited by 2
  5. 2024How Lightweight Can A Vision Transformer Becited by 1
  6. 2025Exploring Active Data Selection Strategies for Continuous Training in Deepfake Detectioncited by 1
  7. 2025SSA3D: Text-Conditioned Assisted Self-Supervised Framework for Automatic Dental Abutment Designcited by 1
  8. 20253D Nephrographic Image Synthesis in CT Urography with the Diffusion Model and Swin Transformercited by 0
  9. 2025Simple Self Organizing Map with Vision Transformerscited by 0
  10. 2025Beyond H&E: Unlocking Pathological Insights with Polarization Imagingcited by 0
  11. 2025Heart Failure Prediction using Modal Decomposition and Masked Autoencoders for Scarce Echocardiography Databasescited by 0
  12. 2025Do You See What I Say? Generalizable Deepfake Detection based on Visual Speech Recognitioncited by 0
5 more
  1. 2026Fusion-SSAT: Unleashing the Potential of Self-supervised Auxiliary Task by Feature Fusion for Generalized Deepfake Detectioncited by 0
  2. 2026CardioMOD-Net: A Modal Decomposition-Neural Network Framework for Diagnosis and Prognosis of HFpEF from Echocardiography Cine Loopscited by 0
  3. 2026How to Build Robust, Scalable Models for GSV-Based Indicators in Neighborhood Researchcited by 0
  4. 2026A Backbone Benchmarking Study on Self-supervised Learning as a Auxiliary Task with Texture-based Local Descriptors for Face Analysiscited by 0
  5. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0