# A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers

arXiv: https://arxiv.org/abs/2508.21148 · HTML: https://mlsanity.com/paper/2508.21148/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 301 of its 1,054 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 301)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)

## Built on by (most cited first, 30 of 37)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing](https://mlsanity.com/paper/2603.09877/) (2026)
- [UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis](https://mlsanity.com/paper/2510.15710/) (2025)
- [SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularization](https://mlsanity.com/paper/2511.17938/) (2025)
- [SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines](https://mlsanity.com/paper/2509.21320/) (2025)
- [When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning](https://mlsanity.com/paper/2605.03314/) (2026)
- [SciDataCopilot: An Agentic Data Preparation Framework for AGI-driven Scientific Discovery](https://mlsanity.com/paper/2602.09132/) (2026)
- [HarmonyCell: Automating Single-Cell Perturbation Modeling under Semantic and Distribution Shifts](https://mlsanity.com/paper/2603.01396/) (2026)
- [Intern-S2-Preview: Scientific Agentic Foundation Model](https://mlsanity.com/paper/2608.13505/) (2026)
- [ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction](https://mlsanity.com/paper/2511.12485/) (2025)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency](https://mlsanity.com/paper/2601.05905/) (2026)
- [AgriWorld:A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agents](https://mlsanity.com/paper/2602.15325/) (2026)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Bidirectional Representations Augmented Autoregressive Biological Sequence Generation](https://mlsanity.com/paper/2510.08169/) (2025)
- [A unified multimodal understanding and generation model for cross-disciplinary scientific research](https://mlsanity.com/paper/2601.01363/) (2026)
- [Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Science](https://mlsanity.com/paper/2603.28361/) (2026)
- [Heterogeneous Scientific Foundation Model Collaboration](https://mlsanity.com/paper/2604.27351/) (2026)
- [Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physics](https://mlsanity.com/paper/2605.17104/) (2026)
- [TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?](https://mlsanity.com/paper/2605.18025/) (2026)
- [AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis](https://mlsanity.com/paper/2510.04997/) (2025)
- [Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs](https://mlsanity.com/paper/2510.23127/) (2025)
- [Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models](https://mlsanity.com/paper/2512.20954/) (2025)
- [Sci-CoE: Co-evolving Scientific Reasoning LLMs via Geometric Consensus with Sparse Supervision](https://mlsanity.com/paper/2602.12164/) (2026)
- [Adversarial Moral Stress Testing of Large Language Models](https://mlsanity.com/paper/2604.01108/) (2026)
- [AblateCell: A Reproduce-then-Ablate Agent for Virtual Cell Repositories](https://mlsanity.com/paper/2604.19606/) (2026)
- [AgentEconomist: An End-to-end Agentic System Translating Economic Intuitions into Executable Computational Experiments](https://mlsanity.com/paper/2604.27725/) (2026)
- [ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoning](https://mlsanity.com/paper/2605.18799/) (2026)
- [Battery-Sim-Agent: Leveraging LLM-Agent for Inverse Battery Parameter Estimation](https://mlsanity.com/paper/2605.29560/) (2026)

## Abstract (arXiv)

Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research, yet their progress is shaped by the complex nature of scientific data. This survey presents a comprehensive, data-centric synthesis that reframes the development of Sci-LLMs as a co-evolution between models and their underlying data substrate. We formulate a unified taxonomy of scientific data and a hierarchical model of scientific knowledge, emphasizing the multimodal, cross-scale, and domain-specific challenges that differentiate scientific corpora from general natural language processing datasets. We systematically review recent Sci-LLMs, from general-purpose foundations to specialized models across diverse scientific disciplines, alongside an extensive analysis of over 270 pre-/post-training datasets, showing why Sci-LLMs pose distinct demands -- heterogeneous, multi-scale, uncertainty-laden corpora that require representations preserving domain invariance and enabling cross-modal reasoning. On evaluation, we examine over 190 benchmark datasets and trace a shift from static exams toward process- and discovery-oriented assessments with advanced evaluation protocols. These data-centric analyses highlight persistent issues in scientific data development and discuss emerging solutions involving semi-automated annotation pipelines and expert validation. Finally, we outline a paradigm shift toward closed-loop systems where autonomous agents based on Sci-LLMs actively experiment, validate, and contribute to a living, evolving knowledge base. Collectively, this work provides a roadmap for building trustworthy, continually evolving artificial intelligence (AI) systems that function as a true partner in accelerating scientific discovery.