# SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

arXiv: https://arxiv.org/abs/2406.09098 · HTML: https://mlsanity.com/paper/2406.09098/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [Predicting Organic Reaction Outcomes with Weisfeiler-Lehman Network](https://mlsanity.com/paper/1709.04555/) (2017)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis](https://mlsanity.com/paper/2403.01976/) (2024)

## Built on by (most cited first, 19 of 19)

- [Training a Scientific Reasoning Model for Chemistry](https://mlsanity.com/paper/2506.17238/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research](https://mlsanity.com/paper/2503.13399/) (2025)
- [VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos](https://mlsanity.com/paper/2506.10857/) (2025)
- [THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models](https://mlsanity.com/paper/2505.22113/) (2025)
- [ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains](https://mlsanity.com/paper/2605.28014/) (2026)
- [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026)
- [WildSci: Advancing Scientific Reasoning from In-the-Wild Literature](https://mlsanity.com/paper/2601.05567/) (2026)
- [CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning](https://mlsanity.com/paper/2510.13166/) (2025)
- [SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning](https://mlsanity.com/paper/2504.02725/) (2025)
- [Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning](https://mlsanity.com/paper/2508.19202/) (2025)
- [Credit Assignment with Resets in Language Model Reasoning](https://mlsanity.com/paper/2605.25507/) (2026)
- [Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following](https://mlsanity.com/paper/2508.02150/) (2025)
- [Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning](https://mlsanity.com/paper/2605.27765/) (2026)
- [A3-Bench: Benchmarking Memory-Driven Scientific Reasoning via Anchor and Attractor Activation](https://mlsanity.com/paper/2601.09274/) (2026)
- [Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning](https://mlsanity.com/paper/2604.27604/) (2026)
- [SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought](https://mlsanity.com/paper/2609.33181/) (2026)
- [From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning](https://mlsanity.com/paper/2610.11373/) (2026)
- [ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry](https://mlsanity.com/paper/2610.11529/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are playing an increasingly important role in scientific research, yet there remains a lack of comprehensive benchmarks to evaluate the breadth and depth of scientific knowledge embedded in these models. To address this gap, we introduce SciKnowEval, a large-scale dataset designed to systematically assess LLMs across five progressive levels of scientific understanding: memory, comprehension, reasoning, discernment, and application. SciKnowEval comprises 28K multi-level questions and solutions spanning biology, chemistry, physics, and materials science. Using this benchmark, we evaluate 20 leading open-source and proprietary LLMs. The results show that while proprietary models often achieve state-of-the-art performance, substantial challenges remain -- particularly in scientific reasoning and real-world application. We envision SciKnowEval as a standard benchmark for evaluating scientific capabilities in LLMs and as a catalyst for advancing more capable and reliable scientific language models.