# How does the pre-training objective affect what large language models learn about linguistic properties? arXiv: https://arxiv.org/abs/2203.10415 ยท HTML: https://mlsanity.com/paper/2203.10415/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 32 references that are in the map. - First posted on arXiv 20 Mar 2022. ## Builds on (most cited first, 23 of 23) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015) - [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019) - [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019) - [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019) - [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018) - [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020) - [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016) - [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021) - [Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models](https://mlsanity.com/paper/2005.10389/) (2020) - [Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing](https://mlsanity.com/paper/2106.02559/) (2021) - [On Losses for Modern Language Models](https://mlsanity.com/paper/2010.01694/) (2020) - [Reservoir Transformers](https://mlsanity.com/paper/2012.15045/) (2020) - [Frustratingly Simple Pretraining Alternatives to Masked Language Modeling](https://mlsanity.com/paper/2109.01819/) (2021) ## Built on by (most cited first, 16 of 16) - [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023) - [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022) - [Large language models in medicine: the potentials and pitfalls](https://mlsanity.com/paper/2309.00087/) (2023) - [E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation](https://mlsanity.com/paper/2205.14912/) (2022) - [Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE](https://mlsanity.com/paper/2302.09268/) (2023) - [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025) - [HashFormers: Towards Vocabulary-independent Pre-trained Transformers](https://mlsanity.com/paper/2210.07904/) (2022) - [Linguistic Blind Spots of Large Language Models](https://mlsanity.com/paper/2503.19260/) (2025) - [Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://mlsanity.com/paper/2601.03448/) (2026) - [How does the task complexity of masked pretraining objectives affect downstream performance?](https://mlsanity.com/paper/2305.10992/) (2023) - [Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?](https://mlsanity.com/paper/2310.17271/) (2023) - [Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Models](https://mlsanity.com/paper/2211.14639/) (2022) - [Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance](https://mlsanity.com/paper/2406.12159/) (2024) - [QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting](https://mlsanity.com/paper/2508.16697/) (2025) - [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026) - [Generate to Understand for Representation](https://mlsanity.com/paper/2306.10056/) (2023) ## Abstract (arXiv) Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our knowledge, no previous work so far has investigated how different pre-training objectives affect what BERT learns about linguistics properties. We hypothesize that linguistically motivated objectives such as MLM should help BERT to acquire better linguistic knowledge compared to other non-linguistically motivated objectives that are not intuitive or hard for humans to guess the association between the input and the label to be predicted. To this end, we pre-train BERT with two linguistically motivated objectives and three non-linguistically motivated ones. We then probe for linguistic characteristics encoded in the representation of the resulting models. We find strong evidence that there are only small differences in probing performance between the representations learned by the two different types of objectives. These surprising results question the dominant narrative of linguistically informed pre-training.