# SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis

arXiv: https://arxiv.org/abs/2403.01976 · HTML: https://mlsanity.com/paper/2403.01976/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 68 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4](https://mlsanity.com/paper/2311.07361/) (2023)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [Large Language Models for Scientific Synthesis, Inference and Explanation](https://mlsanity.com/paper/2310.07984/) (2023)

## Built on by (most cited first, 30 of 41)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research](https://mlsanity.com/paper/2503.13399/) (2025)
- [SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding](https://mlsanity.com/paper/2408.15545/) (2024)
- [A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives](https://mlsanity.com/paper/2503.13415/) (2025)
- [LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs](https://mlsanity.com/paper/2410.14182/) (2024)
- [OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration](https://mlsanity.com/paper/2602.05400/) (2026)
- [SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models](https://mlsanity.com/paper/2406.09098/) (2024)
- [ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models](https://mlsanity.com/paper/2409.13989/) (2024)
- [NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation](https://mlsanity.com/paper/2502.12638/) (2025)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [DeepSeek vs. ChatGPT vs. Claude: A Comparative Study for Scientific Computing and Scientific Machine Learning Tasks](https://mlsanity.com/paper/2502.17764/) (2025)
- [MatTools: Benchmarking Large Language Models for Materials Science Tools](https://mlsanity.com/paper/2505.10852/) (2025)
- [From Generalist to Specialist: A Survey of Large Language Models for Chemistry](https://mlsanity.com/paper/2412.19994/) (2024)
- [BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science](https://mlsanity.com/paper/2407.00466/) (2024)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs](https://mlsanity.com/paper/2505.17139/) (2025)
- [Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learning](https://mlsanity.com/paper/2505.19501/) (2025)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [WildSci: Advancing Scientific Reasoning from In-the-Wild Literature](https://mlsanity.com/paper/2601.05567/) (2026)
- [MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures](https://mlsanity.com/paper/2503.16096/) (2025)
- [CellVerse: Do Large Language Models Really Understand Cell Biology?](https://mlsanity.com/paper/2505.07865/) (2025)
- [Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph](https://mlsanity.com/paper/2405.02105/) (2024)
- [Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts](https://mlsanity.com/paper/2409.13449/) (2024)
- [CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research](https://mlsanity.com/paper/2510.11985/) (2025)
- [HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery](https://mlsanity.com/paper/2512.22899/) (2025)
- [PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality](https://mlsanity.com/paper/2606.20897/) (2026)
- [Doc2SAR: A Synergistic Framework for High-Fidelity Extraction of Structure-Activity Relationships from Scientific Documents](https://mlsanity.com/paper/2506.21625/) (2025)
- [TaxoAlign: Scholarly Taxonomy Generation Using Language Models](https://mlsanity.com/paper/2510.17263/) (2025)
- [Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Models](https://mlsanity.com/paper/2406.05364/) (2024)

## Abstract (arXiv)

Recent breakthroughs in Large Language Models (LLMs) have revolutionized scientific literature analysis. However, existing benchmarks fail to adequately evaluate the proficiency of LLMs in this domain, particularly in scenarios requiring higher-level abilities beyond mere memorization and the handling of multimodal data. In response to this gap, we introduce SciAssess, a benchmark specifically designed for the comprehensive evaluation of LLMs in scientific literature analysis. It aims to thoroughly assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Analysis & Reasoning (L3). It encompasses a variety of tasks drawn from diverse scientific fields, including biology, chemistry, material, and medicine. To ensure the reliability of SciAssess, rigorous quality control measures have been implemented, ensuring accuracy, anonymization, and compliance with copyright standards. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for improvement. We hope this evaluation supports the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are available at urlhttps://github.com/sci-assess/SciAssess.