mlsanityData to 14 Oct 2026

A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models

arXiv 2310.12936 ↗First posted 19 Oct 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. However, multiple underlying factors are associated with an MLM such as its model size, size of the training data, training objectives, the domain from which pretraining data is sampled, tokenization, and languages present in the pretrained corpora, to name a few. It remains unclear as to which of those factors influence social biases that are learned by MLMs. To study the relationship between model factors and the social biases learned by an MLM, as well as the downstream task performance of the model, we conduct a comprehensive study over 39 pretrained MLMs covering different model sizes, training objectives, tokenization methods, training data domains and languages. Our results shed light on important factors often neglected in prior literature, such as tokenization or model objectives.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 59 of its 84 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

59 of 84 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
38 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  5. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2018Neural Network Acceptability Judgmentscited by 958
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  12. 2018Gender Bias in Coreference Resolutioncited by 442
  13. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  14. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  15. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  16. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  17. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  18. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  19. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  20. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  21. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  22. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  23. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 207
  24. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  25. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  26. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  27. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  28. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  29. 2021Debiasing Pre-trained Contextualised Embeddingscited by 107
  30. 2021Sustainable Modular Debiasing of Language Modelscited by 100
  31. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 90
  32. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  33. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  34. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  35. 2020Coreferential Reasoning Learning for Language Representationcited by 80
  36. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  37. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61
  38. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  2. 2025Bias in, Bias out: Annotation Bias in Multilingual Large Language Modelscited by 5
  3. 2024Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Modelscited by 3
  4. 2026SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Culturescited by 0