# Cloze-driven Pretraining of Self-attention Networks

arXiv: https://arxiv.org/abs/1903.07785 · HTML: https://mlsanity.com/paper/1903.07785/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2019.

## Built on by (most cited first, 30 of 41)

- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022)
- [Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning](https://mlsanity.com/paper/2105.03654/) (2021)
- [Domain-matched Pre-training Tasks for Dense Retrieval](https://mlsanity.com/paper/2107.13602/) (2021)
- [DeepStruct: Pretraining of Language Models for Structure Prediction](https://mlsanity.com/paper/2205.10475/) (2022)
- [Self-Influence Guided Data Reweighting for Language Model Pre-training](https://mlsanity.com/paper/2311.00913/) (2023)
- [Learning Rich Representation of Keyphrases from Text](https://mlsanity.com/paper/2112.08547/) (2021)
- [Memorisation versus Generalisation in Pre-trained Language Models](https://mlsanity.com/paper/2105.00828/) (2021)
- [On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study](https://mlsanity.com/paper/2106.00872/) (2021)
- [NeoBERT: A Next-Generation BERT](https://mlsanity.com/paper/2502.19587/) (2025)
- [Capitalization and Punctuation Restoration: a Survey](https://mlsanity.com/paper/2111.10746/) (2021)
- [Domain-Adaptive Pretraining Methods for Dialogue Understanding](https://mlsanity.com/paper/2105.13665/) (2021)
- [Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation](https://mlsanity.com/paper/2104.05964/) (2021)
- [SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation](https://mlsanity.com/paper/2107.14154/) (2021)
- [Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling](https://mlsanity.com/paper/2205.12986/) (2022)
- [Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study](https://mlsanity.com/paper/2304.12269/) (2023)
- [Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss](https://mlsanity.com/paper/2108.07216/) (2021)
- [Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study](https://mlsanity.com/paper/2403.15185/) (2024)
- [Improving speech recognition models with small samples for air traffic control systems](https://mlsanity.com/paper/2102.08015/) (2021)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)
- [Task-adaptive Pre-training of Language Models with Word Embedding Regularization](https://mlsanity.com/paper/2109.08354/) (2021)
- [TMR: Evaluating NER Recall on Tough Mentions](https://mlsanity.com/paper/2103.12312/) (2021)
- [PARSRec: Explainable Personalized Attention-fused Recurrent Sequential Recommendation Using Session Partial Actions](https://mlsanity.com/paper/2209.13015/) (2022)
- [Locally-Contextual Nonlinear CRFs for Sequence Labeling](https://mlsanity.com/paper/2103.16210/) (2021)
- [Sequence-to-Sequence Models for Extracting Information from Registration and Legal Documents](https://mlsanity.com/paper/2201.05658/) (2022)
- [Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?](https://mlsanity.com/paper/2310.17271/) (2023)
- [LLM-DER:A Named Entity Recognition Method Based on Large Language Models for Chinese Coal Chemical Domain](https://mlsanity.com/paper/2409.10077/) (2024)

## Abstract (arXiv)

We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experiments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with the concurrently introduced BERT model. We also present a detailed analysis of a number of factors that contribute to effective pretraining, including data domain and size, model capacity, and variations on the cloze objective.