# OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery

arXiv: https://arxiv.org/abs/2503.17604 · HTML: https://mlsanity.com/paper/2503.17604/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 22 Mar 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text](https://mlsanity.com/paper/2403.18421/) (2024)
- [Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains](https://mlsanity.com/paper/2402.05140/) (2024)
- [MatterChat: A Multi-Modal LLM for Material Science](https://mlsanity.com/paper/2502.13107/) (2025)
- [Adapting Amidst Degradation: Cross Domain Li-ion Battery Health Estimation via Physics-Guided Test-Time Training](https://mlsanity.com/paper/2402.00068/) (2024)
- [Battery State of Health Estimation Using LLM Framework](https://mlsanity.com/paper/2501.18123/) (2025)

## Built on by (most cited first, 20 of 20)

- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [Training a Scientific Reasoning Model for Chemistry](https://mlsanity.com/paper/2506.17238/) (2025)
- [History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL](https://mlsanity.com/paper/2508.18588/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaboration](https://mlsanity.com/paper/2505.15047/) (2025)
- [Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data](https://mlsanity.com/paper/2510.03264/) (2025)
- [RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs](https://mlsanity.com/paper/2510.19225/) (2025)
- [Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learning](https://mlsanity.com/paper/2505.19501/) (2025)
- [WildSci: Advancing Scientific Reasoning from In-the-Wild Literature](https://mlsanity.com/paper/2601.05567/) (2026)
- [Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning](https://mlsanity.com/paper/2508.19202/) (2025)
- [Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Design](https://mlsanity.com/paper/2604.16279/) (2026)
- [JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA](https://mlsanity.com/paper/2605.20284/) (2026)
- [Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard](https://mlsanity.com/paper/2505.13421/) (2025)
- [Training-Free Active Learning Framework in Materials Science with Large Language Models](https://mlsanity.com/paper/2511.19730/) (2025)
- [MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science](https://mlsanity.com/paper/2602.00376/) (2026)
- [RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval](https://mlsanity.com/paper/2506.08625/) (2025)
- [Kunlun Anomaly Troubleshooter: Enabling Kernel-Level Anomaly Detection and Causal Reasoning for Large Model Distributed Inference](https://mlsanity.com/paper/2511.05978/) (2025)
- [A unified multimodal understanding and generation model for cross-disciplinary scientific research](https://mlsanity.com/paper/2601.01363/) (2026)
- [Self-Refining Topology Optimization via an LLM-Based Multi-Agent Framework](https://mlsanity.com/paper/2605.23273/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable potential in advancing scientific knowledge and addressing complex challenges. In this work, we introduce OmniScience, a specialized large reasoning model for general science, developed through three key components: (1) domain adaptive pretraining on a carefully curated corpus of scientific literature, (2) instruction tuning on a specialized dataset to guide the model in following domain-specific tasks, and (3) reasoning-based knowledge distillation through fine-tuning to significantly enhance its ability to generate contextually relevant and logically sound responses. We demonstrate the versatility of OmniScience by developing a battery agent that efficiently ranks molecules as potential electrolyte solvents or additives. Comprehensive evaluations reveal that OmniScience is competitive with state-of-the-art large reasoning models on the GPQA Diamond and domain-specific battery benchmarks, while outperforming all public reasoning and non-reasoning models with similar parameter counts. We further demonstrate via ablation experiments that domain adaptive pretraining and reasoning-based knowledge distillation are critical to attain our performance levels, across benchmarks.