mlsanityData to 14 Oct 2026

Training a Vision Transformer from scratch in less than 24 hours with 1 GPU

arXiv 2211.05187 ↗First posted 9 Nov 2022cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Transformers have become central to recent advances in computer vision. However, training a vision Transformer (ViT) model from scratch can be resource intensive and time consuming. In this paper, we aim to explore approaches to reduce the training costs of ViT models. We introduce some algorithmic improvements to enable training a ViT model from scratch with limited hardware (1 GPU) and time (24 hours) resources. First, we propose an efficient approach to add locality to the ViT architecture. Second, we develop a new image size curriculum learning strategy, which allows to reduce the number of patches extracted from each image at the beginning of the training. Finally, we propose a new variant of the popular ImageNet1k benchmark by adding hardware and time constraints. We evaluate our contributions on this benchmark, and show they can significantly improve performances given the proposed training budget. We will share the code in https://github.com/BorealisAI/efficient-vit-training.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

23 of 34 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2019Searching for MobileNetV3cited by 1,994
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  12. 2021Going deeper with Image Transformerscited by 764
11 more
  1. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  2. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  3. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  4. 2021Anticipative Video Transformercited by 200
  5. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  6. 2019Learning a Deep ConvNet for Multi-label Classification with Partial Labelscited by 124
  7. 2021Vision Transformer for Small-Size Datasetscited by 122
  8. 2021A Survey of Visual Transformerscited by 117
  9. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  10. 2021How to Train BERT with an Academic Budgetcited by 80
  11. 2021A Unified Pruning Framework for Vision Transformerscited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 1; 2026: 1.

6 papers.

  1. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  2. 2024Weight decay induces low-rank attention layerscited by 27
  3. 2023Training Strategies for Vision Transformers for Object Detectioncited by 1
  4. 2024From Coarse to Fine: Efficient Training for Audio Spectrogram Transformerscited by 1
  5. 2025Weakly Supervised Object Segmentation by Background Conditional Divergencecited by 0
  6. 2026The Confusion is Real: GRAPHIC -- A Network Science Approach to Confusion Matrices in Deep Learningcited by 0