mlsanityData to 7 Oct 2026

InforMask: Unsupervised Informative Masking for Language Model Pretraining

arXiv 2210.11771 ↗First posted 21 Oct 2022cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this paper, we propose InforMask, a new unsupervised masking strategy for training masked language models. InforMask exploits Pointwise Mutual Information (PMI) to select the most informative tokens to mask. We further propose two optimizations for InforMask to improve its efficiency. With a one-off preprocessing step, InforMask outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

26 of 39 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
14 more
  1. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  2. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  3. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  4. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  5. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  6. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  7. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  8. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  9. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  10. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  11. 2019SenseBERT: Driving Some Sense into BERTcited by 73
  12. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  13. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  14. 2021Syntax-BERT: Improving Pre-trained Transformers with Syntax Treescited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 3; 2025: 3; 2026: 1.

15 papers.

Sort:

  1. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  2. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  3. 2023Self-Evolution Learning for Discriminative Language Model Pretrainingcited by 8
  4. 2024The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMscited by 7
  5. 2023Unsupervised Improvement of Factual Knowledge in Language Modelscited by 6
  6. 2024EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learningcited by 4
  7. 2023Challenging Decoder helps in Masked Auto-Encoder Pre-training for Dense Passage Retrievalcited by 3
  8. 2023GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learningcited by 2
  9. 2024Unlocking Efficiency: Adaptive Masking for Gene Transformer Modelscited by 2
  10. 2023Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Wayscited by 1
  11. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 1
  12. 2023Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Modelscited by 0
3 more
  1. 2025MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learningcited by 0
  2. 2025Coefficient of Variation Masking: A Volatility-Aware Strategy for EHR Foundation Modelscited by 0
  3. 2026DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classificationcited by 0