mlsanityData to 7 Oct 2026

SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models

arXiv 2307.10635 ↗First posted 20 Jul 2023cs.CL cs.AI cs.LGcited by 246

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Most of the existing Large Language Model (LLM) benchmarks on scientific problem reasoning focus on problems grounded in high-school subjects and are confined to elementary algebraic operations. To systematically examine the reasoning capabilities required for solving complex scientific problems, we introduce an expansive benchmark suite SciBench for LLMs. SciBench contains a carefully curated dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. Based on the dataset, we conduct an in-depth benchmarking study of representative open-source and proprietary LLMs with various prompting strategies. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%. Furthermore, through a detailed user study, we categorize the errors made by LLMs into ten problem-solving abilities. Our analysis indicates that no single prompting strategy significantly outperforms the others and some strategies that demonstrate improvements in certain problem-solving skills could result in declines in other skills. We envision that SciBench will catalyze further developments in the reasoning abilities of LLMs, thereby ultimately contributing to scientific research and discovery.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 64 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

41 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models24 May
  2. 2022A Survey of Deep Learning for Mathematical Reasoning20 Dec
  3. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning29 Sep

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
26 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
  13. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  14. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  15. 2022Large Language Models Can Self-Improvecited by 656
  16. 2022PAL: Program-aided Language Modelscited by 635
  17. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  18. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  19. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  20. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  21. 2023Mathematical Capabilities of ChatGPTcited by 263
  22. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  23. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  24. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  25. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  26. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 47; 2025: 91; 2026: 92.

246 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  4. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  6. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  7. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  8. 2024Tree Search for Language Model Agentscited by 148
  9. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  10. 2024Nemotron-4 340B Technical Reportcited by 143
  11. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  12. 2024Knowledge Fusion of Large Language Modelscited by 110
38 more
  1. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  3. 2025AI-Researcher: Autonomous Scientific Innovationcited by 88
  4. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  5. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  6. 2024Efficient Evolutionary Search Over Chemical Space with Large Language Modelscited by 59
  7. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  8. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  9. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  10. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  11. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  12. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 46
  13. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  14. 2025MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systemscited by 45
  15. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  16. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 42
  17. 2024MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problemscited by 41
  18. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  19. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  20. 2024SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Modelscited by 38
  21. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  22. 2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 37
  23. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  24. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 36
  25. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  26. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  27. 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoningcited by 34
  28. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  29. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 33
  30. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 32
  31. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  32. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
  33. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  34. 2023Structured Chemistry Reasoning with Large Language Modelscited by 27
  35. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  36. 2024Are Large Language Models Good Statisticians?cited by 26
  37. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  38. 2025Base Models Know How to Reason, Thinking Models Learn Whencited by 25

50 most cited of 246 All 246 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain
  2. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoning
  3. 2026SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
  4. 2026GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory
  5. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  6. 2026SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
  7. 2023Large Language Model for Science: A Study on P vs. NP
  8. 2025Evaluation of LLMs for mathematical problem solving