# Frustratingly Simple Pretraining Alternatives to Masked Language Modeling

arXiv: https://arxiv.org/abs/2109.01819 · HTML: https://mlsanity.com/paper/2109.01819/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [Pre-Training Transformers as Energy-Based Cloze Models](https://mlsanity.com/paper/2012.08561/) (2020)
- [On Losses for Modern Language Models](https://mlsanity.com/paper/2010.01694/) (2020)

## Built on by (most cited first, 17 of 17)

- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022)
- [How does the pre-training objective affect what large language models learn about linguistic properties?](https://mlsanity.com/paper/2203.10415/) (2022)
- [Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE](https://mlsanity.com/paper/2302.09268/) (2023)
- [HashFormers: Towards Vocabulary-independent Pre-trained Transformers](https://mlsanity.com/paper/2210.07904/) (2022)
- [Linguistic Blind Spots of Large Language Models](https://mlsanity.com/paper/2503.19260/) (2025)
- [Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://mlsanity.com/paper/2601.03448/) (2026)
- [Instance Regularization for Discriminative Language Model Pre-training](https://mlsanity.com/paper/2210.05471/) (2022)
- [How does the task complexity of masked pretraining objectives affect downstream performance?](https://mlsanity.com/paper/2305.10992/) (2023)
- [GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning](https://mlsanity.com/paper/2307.15933/) (2023)
- [BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition](https://mlsanity.com/paper/2308.08625/) (2023)
- [Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?](https://mlsanity.com/paper/2310.17271/) (2023)
- [Word Boundary Information Isn't Useful for Encoder Language Models](https://mlsanity.com/paper/2401.07923/) (2024)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [Language Model Pre-training on True Negatives](https://mlsanity.com/paper/2212.00460/) (2022)
- [ZhichunRoad at Amazon KDD Cup 2022: MultiTask Pre-Training for E-Commerce Product Search](https://mlsanity.com/paper/2301.13455/) (2023)

## Abstract (arXiv)

Masked language modeling (MLM), a self-supervised pretraining objective, is widely used in natural language processing for learning text representations. MLM trains a model to predict a random sample of input tokens that have been replaced by a [MASK] placeholder in a multi-class setting over the entire vocabulary. When pretraining, it is common to use alongside MLM other auxiliary objectives on the token or sequence level to improve downstream performance (e.g. next sentence prediction). However, no previous work so far has attempted in examining whether other simpler linguistically intuitive or not objectives can be used standalone as main pretraining objectives. In this paper, we explore five simple pretraining objectives based on token-level classification tasks as replacements of MLM. Empirical results on GLUE and SQuAD show that our proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. We further validate our methods using smaller models, showing that pretraining a model with 41% of the BERT-BASE's parameters, BERT-MEDIUM results in only a 1% drop in GLUE scores with our best objective.