mlsanityData to 14 Oct 2026

LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction

arXiv 2411.00177 ↗First posted 31 Oct 2024cond-mat.mtrl-sci cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly being used in materials science. However, little attention has been given to benchmarking and standardized evaluation for LLM-based materials property prediction, which hinders progress. We present LLM4Mat-Bench, the largest benchmark to date for evaluating the performance of LLMs in predicting the properties of crystalline materials. LLM4Mat-Bench contains about 1.9M crystal structures in total, collected from 10 publicly available materials data sources, and 45 distinct properties. LLM4Mat-Bench features different input modalities: crystal composition, CIF, and crystal text description, with 4.7M, 615.5M, and 3.1B tokens in total for each modality, respectively. We use LLM4Mat-Bench to fine-tune models with different sizes, including LLM-Prop and MatBERT, and provide zero-shot and few-shot prompts to evaluate the property prediction capabilities of LLM-chat-like models, including Llama, Gemma, and Mistral. The results highlight the challenges of general-purpose LLMs in materials science and the need for task-specific predictive models and task-specific instruction-tuned LLMs in materials property prediction.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 18 of its 51 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

18 of 51 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  9. 2022Translation between Molecules and Natural Languagecited by 177
  10. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  11. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 93
  12. 2023Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB filescited by 59
6 more
  1. 2024ProLLaMA: A Protein Large Language Model for Multi-Task Protein Language Processingcited by 53
  2. 2023DARWIN Series: Domain Specific Large Language Models for Natural Sciencecited by 39
  3. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillationcited by 32
  4. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  5. 2021Scalable deeper graph neural networks for high-performance materials property predictioncited by 26
  6. 2023CrysMMNet: Multimodal Representation for Crystal Property Predictioncited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 8.

21 papers.

Sort:

By area:

  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  2. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Sciencecited by 15
  3. 2025MatTools: Benchmarking Large Language Models for Materials Science Toolscited by 12
  4. 202534 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discoverycited by 9
  5. 2024Benchmarking large language models for materials synthesis: the case of atomic layer depositioncited by 8
  6. 2025MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Modelscited by 7
  7. 2025A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Toolscited by 7
  8. 2026MADE: Benchmark Environments for Closed-Loop Materials Discoverycited by 7
  9. 2025TopoMAS: Large Language Model Driven Topological Materials Multiagent Systemcited by 6
  10. 2025deCIFer: Crystal Structure Prediction from Powder Diffraction Data using Autoregressive Language Modelscited by 5
  11. 2025AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materialscited by 5
  12. 2025MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledgecited by 3
9 more
  1. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 3
  2. 2025SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Modelscited by 2
  3. 2026Benchmarking bandgap prediction in semiconductors under experimental and realistic evaluation settingscited by 0
  4. 2026Atomistic Language Models Understand and Generate Materialscited by 0
  5. 2026SciMIF: Understanding Multimodal Instruction Following in Scientific Domainscited by 0
  6. 2026DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Predictioncited by 0
  7. 2026Physics as the label for measuring and correcting materials reasoning in multimodal modelscited by 0
  8. 2026CARAT: Do Materials LLMs Reason or Recite?cited by 0
  9. 2026CompMat-Bench: Benchmarking AI Agents for Computational Materials Sciencecited by 0