mlsanityData to 14 Oct 2026

SpanBERT: Improving Pre-training by Representing and Predicting Spans

arXiv 1907.10529 ↗First posted 24 Jul 2019cs.CL cs.LGcited by 834

Not on the map. Why?

Abstract (arXiv)

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT-large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0, respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even show gains on GLUE.

Key facts. Cited by 834 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 225; 2022: 221; 2023: 173; 2024: 113; 2025: 66; 2026: 35.

834 papers.

Sort:

By area:

  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  8. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  9. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  10. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
38 more
  1. 2022UL2: Unifying Language Learning Paradigmscited by 295
  2. 2022Teaching language models to support answers with verified quotescited by 292
  3. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  4. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  5. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  6. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  7. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  8. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  9. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  10. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  11. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  12. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 199
  13. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  14. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  15. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  16. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  17. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  18. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  19. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  20. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
  21. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  22. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 150
  23. 2021Towards Long-Form Video Understandingcited by 146
  24. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  25. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  26. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  27. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  28. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  29. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
  30. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  31. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  32. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  33. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  34. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  35. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
  36. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  37. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  38. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106

50 most cited of 834 All 834 (JSON)