DailyMAE: Towards Pretraining Masked Autoencoders in One Day
Not on the map. Why?
Abstract (arXiv)
Recently, masked image modeling (MIM), an important self-supervised learning (SSL) method, has drawn attention for its effectiveness in learning data representation from unlabeled data. Numerous studies underscore the advantages of MIM, highlighting how models pretrained on extensive datasets can enhance the performance of downstream tasks. However, the high computational demands of pretraining pose significant challenges, particularly within academic environments, thereby impeding the SSL research progress. In this study, we propose efficient training recipes for MIM based SSL that focuses on mitigating data loading bottlenecks and employing progressive training techniques and other tricks to closely maintain pretraining performance. Our library enables the training of a MAE-Base/16 model on the ImageNet 1K dataset for 800 epochs within just 18 hours, using a single machine equipped with 8 A100 GPUs. By achieving speed gains of up to 5.8 times, this work not only demonstrates the feasibility of conducting high-efficiency SSL training but also paves the way for broader accessibility and promotes advancement in SSL research particularly for prototyping and initial testing of SSL ideas. The code is available in https://github.com/erow/FastSSL.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 31 Mar 2024.
Builds on
39 of 46 references.
References · 39
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
27 more
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
- 2017Mixed Precision Trainingcited by 1,135
- 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
- 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
- 2022DeiT III: Revenge of the ViTcited by 417
- 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
- 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
- 2019Fixing the train-test resolution discrepancycited by 209
- 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
- 2021SiT: Self-supervised vIsion Transformercited by 99
- 2023An Inverse Scaling Law for CLIP Trainingcited by 76
- 2022SdAE: Self-distillated Masked Autoencodercited by 52
- 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
- 2023FFCV: Accelerating Training by Removing Data Bottleneckscited by 49
- 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
- 2022Progressive Video Summarization via Multimodal Self-supervised Learningcited by 21
- 2023Towards Democratizing Joint-Embedding Self-Supervised Learningcited by 18
- 2022Object-wise Masked Autoencoders for Fast Pre-trainingcited by 15
- 2021Rethinking Self-Supervised Learning: Small is Beautifulcited by 13
- 2022GMML is All you Needcited by 12
- 2022Siamese Prototypical Contrastive Learningcited by 10
- 2022Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlationcited by 9
- 2022Solving ImageNet: a Unified Scheme for Training any Backbone to Top Resultscited by 8
- 2023Masked Momentum Contrastive Learning for Zero-shot Semantic Understandingcited by 4
- 2023Beyond Accuracy: Statistical Measures and Benchmark for Evaluation of Representation from Self-Supervised Learningcited by 3
Built on by
Citing papers per year: 2024: 2.
2 papers.