mlsanityData to 7 Oct 2026

A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery

arXiv 2406.10833 ↗First posted 16 Jun 2024cs.CLcited by 96

Not on the map. Why?

Cited by 28 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many scientific fields, large language models (LLMs) have revolutionized the way text and other modalities of data (e.g., molecules and proteins) are handled, achieving superior performance in various applications and augmenting the scientific discovery process. Nevertheless, previous surveys on scientific LLMs often concentrate on one or two fields or a single modality. In this paper, we aim to provide a more holistic view of the research landscape by unveiling cross-field and cross-modal connections between scientific LLMs regarding their architectures and pre-training techniques. To this end, we comprehensively survey over 260 scientific LLMs, discuss their commonalities and differences, as well as summarize pre-training datasets and evaluation tasks for each field and modality. Moreover, we investigate how LLMs have been deployed to benefit scientific discovery. Resources related to this survey are available at https://github.com/yuzhimanhua/Awesome-Scientific-Language-Models.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Builds on 239 of its 392 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

239 of 392 references.

References · 239

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  10. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  11. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,768
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2019Invariant Risk Minimizationcited by 1,832
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  13. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  14. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  15. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  16. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  17. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  18. 2019Publicly Available Clinical BERT Embeddingscited by 995
  19. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  20. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  21. 2022Galactica: A Large Language Model for Sciencecited by 736
  22. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  23. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  24. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  25. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  26. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  27. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  28. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  29. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  30. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  31. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  32. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  33. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  34. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  35. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  36. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  37. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  38. 2023Llemma: An Open Language Model For Mathematicscited by 385

50 most cited of 239 All 239 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 55; 2026: 34.

96 papers.

Sort:

By area:

  1. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  2. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  3. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  4. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  5. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  6. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 31
  7. 2025PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaborationcited by 25
  8. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  9. 2025Harnessing Large Language Models for Disaster Management: A Surveycited by 20
  10. 2025LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Modelscited by 20
  11. 2025Protein Large Language Models: A Comprehensive Surveycited by 18
  12. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
38 more
  1. 2025SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Taskscited by 17
  2. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  3. 2025AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmarkcited by 15
  4. 2025How Far Are AI Scientists from Changing the World?cited by 15
  5. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  6. 2025PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priorscited by 13
  7. 2025A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimizationcited by 12
  8. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  9. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  10. 2025SoSBench: Benchmarking Safety Alignment on Six Scientific Domainscited by 11
  11. 2025Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inferencecited by 11
  12. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  13. 2025Toward Greater Autonomy in Materials Discovery Agents: Unifying Planning, Physics, and Scientistscited by 10
  14. 2025Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tablescited by 10
  15. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  16. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  17. 2025Asking a Language Model for Diverse Responsescited by 9
  18. 2024MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Designcited by 8
  19. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  20. 2025AstroAgents: A Multi-Agent AI for Hypothesis Generation from Mass Spectrometry Datacited by 8
  21. 2025mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Moleculescited by 8
  22. 2025Position: The ML Community Must Build an AI-Augmented Peer-Review Ecosystemcited by 8
  23. 2025P1: Mastering Physics Olympiads with Reinforcement Learningcited by 7
  24. 2026P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiadscited by 7
  25. 2026AutoNumerics: An Autonomous, PDE-Agnostic Multi-Agent Pipeline for Scientific Computingcited by 7
  26. 2026Measuring the Gap Between Human and LLM Research Ideascited by 7
  27. 2025PriM: Principle-Inspired Material Discovery through Multi-Agent Collaborationcited by 6
  28. 2025Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discoverycited by 6
  29. 2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMscited by 6
  30. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
  31. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  32. 2024LSHBloom: Memory-efficient, Extreme-scale Document Deduplicationcited by 5
  33. 2025Foundation Models for CPS-IoT: Opportunities and Challengescited by 5
  34. 2025Benchmarking AI scientists for omics data driven biological discoverycited by 5
  35. 2025Advancing AI-Scientist Understanding: Multi-Agent LLMs with Interpretable Physics Reasoningcited by 4
  36. 2025PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Modelscited by 4
  37. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  38. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4

50 most cited of 96 All 96 (JSON)