mlsanityData to 7 Oct 2026

Difference-Masking: Choosing What to Mask in Continued Pretraining

arXiv 2305.14577 ↗First posted 23 May 2023cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The self-supervised objective of masking-and-predicting has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learning outcomes. We investigate this in continued pretraining setting in which pretrained models continue to pretrain on domain-specific data before performing some downstream task. We introduce Difference-Masking, a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. Empirically, we find that Difference-Masking outperforms baselines on continued pretraining settings across four diverse language-only and multimodal video tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 30 of its 57 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

30 of 57 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
18 more
  1. 2016Colorful Image Colorizationcited by 1,113
  2. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
  3. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  4. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  6. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  7. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  8. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  9. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  10. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 113
  11. 2018Object Level Visual Reasoning in Videoscited by 67
  12. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  13. 2020Train No Evil: Selective Masking for Task-Guided Pre-Trainingcited by 36
  14. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  15. 2023Salient Span Masking for Temporal Understandingcited by 14
  16. 2022AANG: Automating Auxiliary Learningcited by 12
  17. 2020Studying Strategically: Learning to Mask for Closed-book QAcited by 10
  18. 2022Face-to-Face Contrastive Learning for Social Intelligence Question-Answeringcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Efficient Vision-Language Pre-training by Cluster Maskingcited by 13
  2. 2024KidLM: Advancing Language Models for Children -- Early Insights and Future Directionscited by 7
  3. 2025Cognitive-Aligned Spatio-Temporal Large Language Models For Next Point-of-Interest Predictioncited by 2
  4. 2026KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Trainingcited by 0