# SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

arXiv: https://arxiv.org/abs/2308.13149 · HTML: https://mlsanity.com/paper/2308.13149/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies](https://mlsanity.com/paper/2212.08104/) (2022)

## Built on by (most cited first, 30 of 115)

- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [MAmmoTH2: Scaling Instructions from the Web](https://mlsanity.com/paper/2405.03548/) (2024)
- [A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist](https://mlsanity.com/paper/2402.18485/) (2024)
- [MMVU: Measuring Expert-Level Multi-Discipline Video Understanding](https://mlsanity.com/paper/2501.12380/) (2025)
- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI](https://mlsanity.com/paper/2406.12753/) (2024)
- [SciAgent: Tool-augmented Language Models for Scientific Reasoning](https://mlsanity.com/paper/2402.11451/) (2024)
- [PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning](https://mlsanity.com/paper/2502.12054/) (2025)
- [HoneyComb: A Flexible LLM-Based Agent System for Materials Science](https://mlsanity.com/paper/2409.00135/) (2024)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Developing ChemDFM as a large language foundation model for chemistry](https://mlsanity.com/paper/2401.14818/) (2024)
- [SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis](https://mlsanity.com/paper/2403.01976/) (2024)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning](https://mlsanity.com/paper/2501.06590/) (2025)
- [SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models](https://mlsanity.com/paper/2401.07950/) (2024)
- [Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models](https://mlsanity.com/paper/2505.03469/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs](https://mlsanity.com/paper/2505.16997/) (2025)
- [EXP-Bench: Can AI Conduct AI Research Experiments?](https://mlsanity.com/paper/2505.24785/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Are large language models superhuman chemists?](https://mlsanity.com/paper/2404.01475/) (2024)
- [SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning](https://mlsanity.com/paper/2505.19099/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents](https://mlsanity.com/paper/2502.16069/) (2025)
- [Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving](https://mlsanity.com/paper/2503.21821/) (2025)
- [LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs](https://mlsanity.com/paper/2410.14182/) (2024)

## Abstract (arXiv)

Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The codes and data are publicly available on https://github.com/OpenDFM/SciEval.