mlsanityData to 7 Oct 2026

A Survey of Pre-trained Language Models for Processing Scientific Text

arXiv 2401.17824 ↗First posted 31 Jan 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

The number of Language Models (LMs) dedicated to processing scientific text is on the rise. Keeping pace with the rapid growth of scientific LMs (SciLMs) has become a daunting task for researchers. To date, no comprehensive surveys on SciLMs have been undertaken, leaving this issue unaddressed. Given the constant stream of new SciLMs, appraising the state-of-the-art and how they compare to each other remain largely unknown. This work fills that gap and provides a comprehensive review of SciLMs, including an extensive analysis of their effectiveness across different domains, tasks and datasets, and a discussion on the challenges that lie ahead.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 73 of its 263 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

73 of 263 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  13. 2019Publicly Available Clinical BERT Embeddingscited by 995
  14. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  15. 2022Galactica: A Large Language Model for Sciencecited by 736
  16. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  17. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  18. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  19. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  20. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  21. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  22. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  23. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  24. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  25. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  26. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  27. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  28. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  29. 2020Language Models are Open Knowledge Graphscited by 95
  30. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 89
  31. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 85
  32. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  33. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  34. 2023K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilizationcited by 69
  35. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
  36. 2021MathBERT: A Pre-Trained Model for Mathematical Formula Understandingcited by 58
  37. 2022CSL: A Large-scale Chinese Scientific Literature Datasetcited by 45
  38. 2021MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Educationcited by 37

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2025Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectivescited by 2
  3. 2024Towards Efficient Large Language Models for Scientific Text: A Reviewcited by 0
  4. 2024Towards understanding evolution of science through language model seriescited by 0
  5. 2025Enhancing Recommender Systems Using Textual Embeddings from Pre-trained Language Modelscited by 0
  6. 2026Entropy-Based Data Selection for Language Modelscited by 0
  7. 2026KeySI: An Interaction Framework for Tuning Text Embeddings Based on Human Feedbackcited by 0