mlsanityData to 7 Oct 2026

On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias

arXiv 2110.05301 ↗First posted 11 Oct 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite the success of pretrained masked language models (MLM), why MLM pretraining is useful is still a qeustion not fully answered. In this work we theoretically and empirically show that MLM pretraining makes models robust to lexicon-level spurious features, partly answer the question. We theoretically show that, when we can model the distribution of a spurious feature Pi conditioned on the context, then (1) Pi is at least as informative as the spurious feature, and (2) learning from Pi is at least as simple as learning from the spurious feature. Therefore, MLM pretraining rescues the model from the simplicity bias caused by the spurious feature. We also explore the efficacy of MLM pretraing in causal settings. Finally we close the gap between our theories and the real world practices by conducting experiments on the hate speech detection and the name entity recognition tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

34 of 41 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  4. 2018MINE: Mutual Information Neural Estimationcited by 680
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  7. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  9. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  10. 2019Measuring Bias in Contextualized Word Representationscited by 275
  11. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  12. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
22 more
  1. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  2. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  3. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  4. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  5. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  6. 2020Invariant Rationalizationcited by 131
  7. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  8. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  9. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  10. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  11. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  12. 2020Contextualizing Hate Speech Classifiers with Post-hoc Explanationcited by 84
  13. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  14. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71
  15. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  16. 2020Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learningcited by 44
  17. 2021Understanding Hard Negatives in Noise Contrastive Estimationcited by 39
  18. 2020Exploring the Linear Subspace Hypothesis in Gender Bias Mitigationcited by 25
  19. 2020Investigating representations of verb bias in neural language modelscited by 22
  20. 2020Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddingscited by 20
  21. 2021On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependenciescited by 18
  22. 2020A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?cited by 6

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1.

4 papers.

  1. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  2. 2025Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategiescited by 8
  3. 2022Enhancing Tabular Reasoning with Pattern Exploiting Trainingcited by 3
  4. 2023The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretrainingcited by 1