Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning
Not on the map. Why?
Abstract (arXiv)
In this paper, we present an innovative approach to self-supervised learning for Vision Transformers (ViTs), integrating local masked image modeling with progressive layer freezing. This method focuses on enhancing the efficiency and speed of initial layer training in ViTs. By systematically freezing specific layers at strategic points during training, we reduce computational demands while maintaining or improving learning capabilities. Our approach employs a novel multi-scale reconstruction process that fosters efficient learning in initial layers and enhances semantic comprehension across scales. The results demonstrate a substantial reduction in training time (~12.5%) with a minimal impact on model accuracy (decrease in top-1 accuracy by 0.6%). Our method achieves top-1 and top-5 accuracies of 82.6% and 96.2%, respectively, underscoring its potential in scenarios where computational resources and time are critical. This work marks an advancement in the field of self-supervised learning for computer vision. The implementation of our approach is available at our project's GitHub repository: github.com/utkutpcgl/ViTFreeze.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 2 Dec 2023.
Builds on
34 of 43 references.
References · 34
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
- 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
22 more
- 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
- 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
- 2017Colorization as a Proxy Task for Visual Understandingcited by 175
- 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
- 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
- 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
- 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
- 2017FreezeOut: Accelerate Training by Progressively Freezing Layerscited by 54
- 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
- 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
- 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 47
- 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
- 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
- 2023Efficient Post-training Quantization with FP8 Formatscited by 29
- 2023Pruning Deep Neural Networks from a Sparsity Perspectivecited by 25
- 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
- 2021Self-Adaptive Training: Bridging Supervised and Self-Supervised Learningcited by 17
- 2023M2T: Masking Transformers Twice for Faster Decodingcited by 14
- 2022FastMIM: Expediting Masked Image Modeling Pre-training for Visioncited by 7
- 2022Disjoint Masking with Joint Distillation for Efficient Masked Image Modelingcited by 6
- 2023Efficient Self-supervised Continual Learning with Progressive Task-correlated Layer Freezingcited by 6
- 2023Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipelinecited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.