mlsanityData to 7 Oct 2026

Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training

arXiv 2404.08327 ↗First posted 12 Apr 2024cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for `tailored' masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

35 of 43 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
23 more
  1. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  2. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  3. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  4. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  5. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  6. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  7. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  8. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 113
  9. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  10. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  11. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  12. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  13. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  14. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  15. 2022Masked Image Modeling with Denoising Contrastcited by 45
  16. 2022How to Understand Masked Autoencoderscited by 37
  17. 2022A Unified View of Masked Image Modelingcited by 36
  18. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  19. 2022Attentive Mask CLIPcited by 23
  20. 2022Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscited by 18
  21. 2022Object-wise Masked Autoencoders for Fast Pre-trainingcited by 15
  22. 2022Exploring the Coordination of Frequency and Attention in Masked Image Modelingcited by 12
  23. 2022Efficient Pre-training of Masked Language Model via Concept-based Curriculum Maskingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 2; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  2. 2024Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detectioncited by 5
  3. 2025TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learningcited by 5
  4. 2024SG-MIM: Structured Knowledge Guided Efficient Pre-training for Dense Predictioncited by 4
  5. 2024UniTT-Stereo: Unified Training of Transformer for Enhanced Stereo Matchingcited by 3
  6. 2024iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptationcited by 3
  7. 2025Masked Diffusion Language Models with Frequency-Informed Trainingcited by 2
  8. 2026DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentationcited by 2
  9. 2024MaDis-Stereo: Enhanced Stereo Matching via Distilled Masked Image Modelingcited by 1
  10. 2026AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformerscited by 1
  11. 2024Collaborative Learning for Enhanced Unsupervised Domain Adaptationcited by 0
  12. 2026Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervisioncited by 0
2 more
  1. 2026Isotropic Embedding Perturbations for Robust Vision Language Encoderscited by 0
  2. 2026Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learningcited by 0