mlsanityData to 14 Oct 2026

SciBERT: A Pretrained Language Model for Scientific Text

arXiv 1903.10676 ↗First posted 26 Mar 2019cs.CLcited by 1,414

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.

Key facts. Cited by 1,414 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 216; 2022: 228; 2023: 310; 2024: 281; 2025: 233; 2026: 145.

1,414 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2023BloombergGPT: A Large Language Model for Financecited by 939
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  6. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  7. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  8. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  9. 2023Llemma: An Open Language Model For Mathematicscited by 388
  10. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  11. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  12. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
38 more
  1. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  2. 2023Continual Pre-training of Language Modelscited by 198
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  4. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  5. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  6. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  7. 2022Translation between Molecules and Natural Languagecited by 177
  8. 2023GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networkscited by 167
  9. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 166
  10. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 149
  11. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  12. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  13. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
  14. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  15. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  16. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  17. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  18. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 123
  19. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  20. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 120
  21. 2023The Semantic Scholar Open Data Platformcited by 118
  22. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 118
  23. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  24. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  25. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  26. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  27. 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
  28. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  29. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  30. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 101
  31. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  32. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  33. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  34. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  35. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 95
  36. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  37. 2024Tuning Language Models by Proxycited by 89
  38. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87

50 most cited of 1,414 All 1,414 (JSON)