mlsanityData to 14 Oct 2026

Cloze-driven Pretraining of Self-attention Networks

arXiv 1903.07785 ↗First posted 19 Mar 2019cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experiments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with the concurrently introduced BERT model. We also present a detailed analysis of a number of factors that contribute to effective pretraining, including data domain and size, model capacity, and variations on the cloze objective.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 21; 2022: 10; 2023: 5; 2024: 4; 2025: 1.

41 papers.

Sort:

By area:

  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  2. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  3. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  4. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  5. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  6. 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
  7. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  8. 2022DeepStruct: Pretraining of Language Models for Structure Predictioncited by 45
  9. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  10. 2021Learning Rich Representation of Keyphrases from Textcited by 36
  11. 2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32
  12. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 31
29 more
  1. 2025NeoBERT: A Next-Generation BERTcited by 16
  2. 2021Capitalization and Punctuation Restoration: a Surveycited by 13
  3. 2021Domain-Adaptive Pretraining Methods for Dialogue Understandingcited by 11
  4. 2021Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translationcited by 9
  5. 2021SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluationcited by 8
  6. 2022Transcormer: Transformer for Sentence Scoring with Sliding Language Modelingcited by 8
  7. 2023Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Studycited by 8
  8. 2021Partially Supervised Named Entity Recognition via the Expected Entity Ratio Losscited by 7
  9. 2024Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Studycited by 7
  10. 2021Improving speech recognition models with small samples for air traffic control systemscited by 6
  11. 2021Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speechcited by 6
  12. 2021Task-adaptive Pre-training of Language Models with Word Embedding Regularizationcited by 5
  13. 2021TMR: Evaluating NER Recall on Tough Mentionscited by 4
  14. 2022PARSRec: Explainable Personalized Attention-fused Recurrent Sequential Recommendation Using Session Partial Actionscited by 4
  15. 2021Locally-Contextual Nonlinear CRFs for Sequence Labelingcited by 2
  16. 2022Sequence-to-Sequence Models for Extracting Information from Registration and Legal Documentscited by 2
  17. 2023Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?cited by 2
  18. 2024LLM-DER:A Named Entity Recognition Method Based on Large Language Models for Chinese Coal Chemical Domaincited by 2
  19. 2021HLE-UPC at SemEval-2021 Task 5: Multi-Depth DistilBERT for Toxic Spans Detectioncited by 1
  20. 2021Analysing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasetscited by 1
  21. 2021Identification of Fine-Grained Location Mentions in Crisis Tweetscited by 1
  22. 2022Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languagescited by 1
  23. 2022Fast Rule-Based Decoding: Revisiting Syntactic Rules in Neural Constituency Parsingcited by 1
  24. 2021Data Centric Domain Adaptation for Historical Text with OCR Errorscited by 0
  25. 2022Multilinguals at SemEval-2022 Task 11: Transformer Based Architecture for Complex NERcited by 0
  26. 2022Spontaneous Emerging Preference in Two-tower Language Modelcited by 0
  27. 2023An Anchor Learning Approach for Citation Field Learningcited by 0
  28. 2024Fine Tuning Named Entity Extraction Models for the Fantasy Domaincited by 0
  29. 2024A Brief History of Named Entity Recognitioncited by 0