mlsanityData to 14 Oct 2026

SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

arXiv 2308.13149 ↗First posted 25 Aug 2023cs.CLcited by 115

Not on the map. Why?

Cited by 30 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The codes and data are publicly available on https://github.com/OpenDFM/SciEval.

Key facts. Cited by 115 papers in the map (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

20 of 32 references.

References · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
8 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  3. 2022Galactica: A Large Language Model for Sciencecited by 744
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  6. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  7. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  8. 2022The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategiescited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 32; 2025: 47; 2026: 31.

115 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2025s1: Simple test-time scalingcited by 1,341
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  6. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  7. 2024A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalistcited by 137
  8. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  9. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  10. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 71
  11. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 70
  12. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 59
38 more
  1. 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Sciencecited by 57
  2. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  3. 2024Developing ChemDFM as a large language foundation model for chemistrycited by 52
  4. 2024SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysiscited by 41
  5. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 41
  6. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  7. 2024SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Modelscited by 39
  8. 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Modelscited by 39
  9. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  10. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 38
  11. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  12. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  13. 2024Are large language models superhuman chemists?cited by 34
  14. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 34
  15. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  16. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 33
  17. 2025Physics: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  18. 2024LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labscited by 25
  19. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 25
  20. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agentscited by 24
  21. 2024Watermarking Techniques for Large Language Models: A Surveycited by 23
  22. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  23. 2024SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Modelscited by 19
  24. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  25. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 17
  26. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Webcited by 16
  27. 2024Towards Effective and Efficient Continual Pre-training of Large Language Modelscited by 15
  28. 2025SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Modelscited by 15
  29. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Sciencecited by 15
  30. 2026Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoningcited by 15
  31. 2023PsyEval: A Suite of Mental Health Related Tasks for Evaluating Large Language Modelscited by 13
  32. 2024Generative Hierarchical Materials Searchcited by 13
  33. 2025MatTools: Benchmarking Large Language Models for Materials Science Toolscited by 12
  34. 2024SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Typescited by 10
  35. 2025MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problemscited by 10
  36. 2025U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understandingcited by 10
  37. 2025NetArena: Dynamic Benchmarks for AI Agents in Network Automationcited by 9
  38. 2025Can Theoretical Physics Research Benefit from Language Agents?cited by 9

50 most cited of 115 All 115 (JSON)