# On Losses for Modern Language Models

arXiv: https://arxiv.org/abs/2010.01694 · HTML: https://mlsanity.com/paper/2010.01694/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives](https://mlsanity.com/paper/2102.12982/) (2021)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative](https://mlsanity.com/paper/2109.07437/) (2021)
- [Digger: Detecting Copyright Content Mis-usage in Large Language Model Training](https://mlsanity.com/paper/2401.00676/) (2024)
- [Frustratingly Simple Pretraining Alternatives to Masked Language Modeling](https://mlsanity.com/paper/2109.01819/) (2021)
- [Assemble Foundation Models for Automatic Code Summarization](https://mlsanity.com/paper/2201.05222/) (2022)
- [How does the pre-training objective affect what large language models learn about linguistic properties?](https://mlsanity.com/paper/2203.10415/) (2022)
- [Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization](https://mlsanity.com/paper/2210.09175/) (2022)
- [Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling](https://mlsanity.com/paper/2210.05043/) (2022)
- [Building Chinese Biomedical Language Models via Multi-Level Text Discrimination](https://mlsanity.com/paper/2110.07244/) (2021)
- [Training ELECTRA Augmented with Multi-word Selection](https://mlsanity.com/paper/2106.00139/) (2021)
- [Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://mlsanity.com/paper/2601.03448/) (2026)
- [An Experimental Study on Pretraining Transformers from Scratch for IR](https://mlsanity.com/paper/2301.10444/) (2023)
- [How does the task complexity of masked pretraining objectives affect downstream performance?](https://mlsanity.com/paper/2305.10992/) (2023)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [A State-Vector Framework for Dataset Effects](https://mlsanity.com/paper/2310.10955/) (2023)
- [Training self-supervised peptide sequence models on artificially chopped proteins](https://mlsanity.com/paper/2211.06428/) (2022)
- [POSPAN: Position-Constrained Span Masking for Language Model Pre-training](https://mlsanity.com/paper/2609.16061/) (2026)

## Abstract (arXiv)

BERT set many state-of-the-art results over varied NLU benchmarks by pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP), the latter of which has been highly criticized. In this paper, we 1) clarify NSP's effect on BERT pre-training, 2) explore fourteen possible auxiliary pre-training tasks, of which seven are novel to modern language models, and 3) investigate different ways to include multiple tasks into pre-training. We show that NSP is detrimental to training due to its context splitting and shallow semantic signal. We also identify six auxiliary pre-training tasks -- sentence ordering, adjacent sentence prediction, TF prediction, TF-IDF prediction, a FastSent variant, and a Quick Thoughts variant -- that outperform a pure MLM baseline. Finally, we demonstrate that using multiple tasks in a multi-task pre-training framework provides better results than using any single auxiliary task. Using these methods, we outperform BERT Base on the GLUE benchmark using fewer than a quarter of the training tokens.