mlsanityData to 14 Oct 2026

Disjoint Masking with Joint Distillation for Efficient Masked Image Modeling

arXiv 2301.00230 ↗First posted 31 Dec 2022cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Masked image modeling (MIM) has shown great promise for self-supervised learning (SSL) yet been criticized for learning inefficiency. We believe the insufficient utilization of training signals should be responsible. To alleviate this issue, we introduce a conceptually simple yet learning-efficient MIM training scheme, termed Disjoint Masking with Joint Distillation (DMJD). For disjoint masking (DM), we sequentially sample multiple masked views per image in a mini-batch with the disjoint regulation to raise the usage of tokens for reconstruction in each image while keeping the masking rate of each view. For joint distillation (JD), we adopt a dual branch architecture to respectively predict invisible (masked) and visible (unmasked) tokens with superior learning targets. Rooting in orthogonal perspectives for training efficiency improvement, DM and JD cooperatively accelerate the training convergence yet not sacrificing the model generalization ability. Concretely, DM can train ViT with half of the effective training epochs (3.7 times less time-consuming) to report competitive performance. With JD, our DMJD clearly improves the linear probing classification accuracy over ConvMAE by 5.8%. On fine-grained downstream tasks like semantic segmentation, object detection, etc., our DMJD also presents superior generalization compared with state-of-the-art SSL methods. The code and model will be made public at https://github.com/mx-mark/DMJD.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 31 Dec 2022.

Builds on

55 of 67 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Mask R-CNNcited by 7,277
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2016Layer Normalizationcited by 5,669
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  10. 2021Going deeper with Image Transformerscited by 764
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  13. 2017Large Batch Training of Convolutional Networkscited by 535
  14. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  15. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  16. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  17. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  18. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  19. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  20. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  21. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  22. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  23. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  24. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  25. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  26. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  27. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  28. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  29. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  30. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  31. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  32. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  33. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  34. 2022SdAE: Self-distillated Masked Autoencodercited by 52
  35. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  36. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 33
  37. 2022mc-BEiT: Multi-choice Discretization for Image BERT Pre-trainingcited by 31
  38. 2022Efficient Self-supervised Vision Pretraining with Local Masked Reconstructioncited by 27

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 2.

6 papers.

  1. 2023CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentationcited by 15
  2. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  3. 2025Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognitioncited by 6
  4. 2023Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learningcited by 1
  5. 2024StyleInject: Parameter Efficient Tuning of Text-to-Image Diffusion Modelscited by 1
  6. 2025Automated Learning of Semantic Embedding Representations for Diffusion Modelscited by 0