mlsanityData to 14 Oct 2026

MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge

arXiv 2505.23982 ↗First posted 29 May 2025cs.AIcited by 3

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Despite recent advances in large language models (LLMs) for materials science, there is a lack of benchmarks for evaluating their domain-specific knowledge and complex reasoning abilities. To bridge this gap, we introduce MSQA, a comprehensive evaluation benchmark of 1,757 graduate-level materials science questions in two formats: detailed explanatory responses and binary True/False assessments. MSQA distinctively challenges LLMs by requiring both precise factual knowledge and multi-step reasoning across seven materials science sub-fields, such as structure-property relationships, synthesis processes, and computational modeling. Through experiments with 10 state-of-the-art LLMs, we identify significant gaps in current LLM performance. While API-based proprietary LLMs achieve up to 84.5% accuracy, open-source (OSS) LLMs peak around 60.5%, and domain-specific LLMs often underperform significantly due to overfitting and distributional shifts. MSQA represents the first benchmark to jointly evaluate the factual and reasoning capabilities of LLMs crucial for LLMs in advanced materials science.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 29 of its 47 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

29 of 47 references.

References · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  10. 2024Phi-4 Technical Reportcited by 711
  11. 2022Large Language Models Can Self-Improvecited by 664
  12. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
17 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  2. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  3. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  4. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  5. 2022Holistic Evaluation of Language Modelscited by 131
  6. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 128
  7. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
  8. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  9. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  10. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  11. 2023MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modelingcited by 35
  12. 2025LLMs Can Generate a Better Answer by Aggregating Their Own Responsescited by 27
  13. 2023HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Sciencecited by 22
  14. 2024LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Predictioncited by 21
  15. 2023PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literaturecited by 13
  16. 2023MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Modelscited by 11
  17. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictionscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026TritonDFT: Automating DFT with a Multi-Agent Frameworkcited by 5
  2. 2026PolyReal: A Benchmark for Real-World Polymer Science Workflowscited by 1
  3. 2026OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfieldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
  2. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation
  3. 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Science
  4. 202534 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discovery
  5. 2025Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents
  6. 2025MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis
  7. 2026From Knowledge to Action: Outcomes of the 2025 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry
  8. 2025Role of Large Language Models and Retrieval-Augmented Generation for Accelerating Crystalline Material Discovery: A Systematic Review