mlsanityData to 7 Oct 2026

SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations

arXiv 2308.08884 ↗First posted 17 Aug 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Due to the prevalence of scale variance in nature images, we propose to use image scale as a self-supervised signal for Masked Image Modeling (MIM). Our method involves selecting random patches from the input image and downsampling them to a low-resolution format. Our framework utilizes the latest advances in super-resolution (SR) to design the prediction head, which reconstructs the input from low-resolution clues and other patches. After 400 epochs of pre-training, our Super Resolution Masked Autoencoders (SRMAE) get an accuracy of 82.1% on the ImageNet-1K task. Image scale signal also allows our SRMAE to capture scale invariance representation. For the very low resolution (VLR) recognition task, our model achieves the best performance, surpassing DeriveNet by 1.3%. Our method also achieves an accuracy of 74.84% on the task of recognizing low-resolution facial expressions, surpassing the current state-of-the-art FMD by 9.48%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

54 of 79 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2017Enhanced Deep Residual Networks for Single Image Super-Resolutioncited by 1,543
38 more
  1. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  2. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  3. 2018Image Super-Resolution Using Very Deep Residual Channel Attention Networkscited by 1,256
  4. 2014Image Super-Resolution Using Deep Convolutional Networkscited by 1,218
  5. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2020Pre-Trained Image Processing Transformercited by 891
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  10. 2019Relational Knowledge Distillationcited by 873
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  13. 2019Similarity-Preserving Knowledge Distillationcited by 507
  14. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  15. 2019Variational Information Distillation for Knowledge Transfercited by 321
  16. 2019Correlation Congruence for Knowledge Distillationcited by 280
  17. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  18. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  19. 2019Lightweight Image Super-Resolution with Information Multi-distillation Networkcited by 227
  20. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  21. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  22. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  23. 2021Transformer for Single Image Super-Resolutioncited by 150
  24. 2018Fast and Accurate Single Image Super-Resolution via Information Distillation Networkcited by 121
  25. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  26. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  27. 2018Low-resolution Face Recognition in the Wild via Selective Knowledge Distillationcited by 55
  28. 2016From Facial Expression Recognition to Interpersonal Relation Predictioncited by 53
  29. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  30. 2018Super-Identity Convolutional Neural Network for Face Hallucinationcited by 39
  31. 2016Studying Very Low Resolution Recognition Using Deep Networkscited by 36
  32. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  33. 2020Robust Re-Identification by Multiple Views Knowledge Distillationcited by 29
  34. 2022Efficient Self-supervised Vision Pretraining with Local Masked Reconstructioncited by 27
  35. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  36. 2021Multi-Scale Aligned Distillation for Low-Resolution Detectioncited by 24
  37. 2018Face hallucination using cascaded super-resolution and identity priorscited by 22
  38. 2019Cross-Resolution Learning for Face Recognitioncited by 22

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.