mlsanityData to 14 Oct 2026

How Well Do Vision Transformers (VTs) Transfer To The Non-Natural Image Domain? An Empirical Study Involving Art Classification

arXiv 2208.04693 ↗First posted 9 Aug 2022cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (VTs) are becoming a valuable alternative to Convolutional Neural Networks (CNNs) when it comes to problems involving high-dimensional and spatially organized inputs such as images. However, their Transfer Learning (TL) properties are not yet well studied, and it is not fully known whether these neural architectures can transfer across different domains as well as CNNs. In this paper we study whether VTs that are pre-trained on the popular ImageNet dataset learn representations that are transferable to the non-natural image domain. To do so we consider three well-studied art classification problems and use them as a surrogate for studying the TL potential of four popular VTs. Their performance is extensively compared against that of four common CNNs across several TL experiments. Our results show that VTs exhibit strong generalization properties and that these networks are more powerful feature extractors than CNNs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 9 Aug 2022.

Builds on

25 of 46 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  11. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
13 more
  1. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
  2. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  3. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  4. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  5. 2021TransFER: Learning Relation-aware Facial Expression Representations with Transformerscited by 85
  6. 2020Deep-COVID: Predicting COVID-19 From Chest X-Ray Images Using Deep Transfer Learningcited by 67
  7. 2021Is it Time to Replace CNNs with Transformers for Medical Images?cited by 60
  8. 2021ConvNets vs. Transformers: Whose Visual Representations are More Transferable?cited by 35
  9. 2021Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generationcited by 30
  10. 2018Weakly Supervised Object Detection in Artworkscited by 27
  11. 2020A Data Set and a Convolutional Model for Iconography Classification in Paintingscited by 17
  12. 2017OmniArt: Multi-task Deep Learning for Artistic Data Analysiscited by 16
  13. 2018Using transfer learning to detect galaxy mergerscited by 11

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2023ARTxAI: Explainable Artificial Intelligence Curates Deep Representation Learning for Artistic Images using Fuzzy Techniquescited by 10
  2. 2024NT-VOT211: A Large-Scale Benchmark for Night-time Visual Object Trackingcited by 3
  3. 2025Frequency Domain Enhanced U-Net for Low-Frequency Information-Rich Image Segmentation in Surgical and Deep-Sea Exploration Robotscited by 1