# SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery

arXiv: https://arxiv.org/abs/2509.08032 · HTML: https://mlsanity.com/paper/2509.08032/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 18 of its 19 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 18 of 18)

- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [FinGPT: Democratizing Internet-scale Data for Financial Large Language Models](https://mlsanity.com/paper/2307.10485/) (2023)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners](https://mlsanity.com/paper/2204.07689/) (2022)
- [Large Language Models are Zero Shot Hypothesis Proposers](https://mlsanity.com/paper/2311.05965/) (2023)
- [SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding](https://mlsanity.com/paper/2408.15545/) (2024)
- [Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization](https://mlsanity.com/paper/2402.12550/) (2024)
- [SciPIP: An LLM-based Scientific Paper Idea Proposer](https://mlsanity.com/paper/2410.23166/) (2024)
- [Automated Review Generation Method Based on Large Language Models](https://mlsanity.com/paper/2407.20906/) (2024)

## Abstract (arXiv)

Scientific literature is growing exponentially, creating a critical bottleneck for researchers to efficiently synthesize knowledge. While general-purpose Large Language Models (LLMs) show potential in text processing, they often fail to capture scientific domain-specific nuances (e.g., technical jargon, methodological rigor) and struggle with complex scientific tasks, limiting their utility for interdisciplinary research. To address these gaps, this paper presents SciGPT, a domain-adapted foundation model for scientific literature understanding and ScienceBench, an open source benchmark tailored to evaluate scientific LLMs. Built on the Qwen3 architecture, SciGPT incorporates three key innovations: (1) low-cost domain distillation via a two-stage pipeline to balance performance and efficiency; (2) a Sparse Mixture-of-Experts (SMoE) attention mechanism that cuts memory consumption by 55% for 32,000-token long-document reasoning; and (3) knowledge-aware adaptation integrating domain ontologies to bridge interdisciplinary knowledge gaps. Experimental results on ScienceBench show that SciGPT outperforms GPT-4o in core scientific tasks including sequence labeling, generation, and inference. It also exhibits strong robustness in unseen scientific tasks, validating its potential to facilitate AI-augmented scientific discovery.