mlsanityData to 14 Oct 2026

MatterChat: A Multi-Modal LLM for Material Science

arXiv 2502.13107 ↗First posted 18 Feb 2025cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Understanding and predicting the properties of inorganic materials is crucial for accelerating advancements in materials science and driving applications in energy, electronics, and beyond. Integrating material structure data with language-based information through multi-modal large language models (LLMs) offers great potential to support these efforts by enhancing human-AI interaction. However, a key challenge lies in integrating atomic structures at full resolution into LLMs. In this work, we introduce MatterChat, a versatile structure-aware multi-modal LLM that unifies material structural data and textual inputs into a single cohesive model. MatterChat employs a bridging module to effectively align a pretrained machine learning interatomic potential with a pretrained LLM, reducing training costs and enhancing flexibility. Our results demonstrate that MatterChat significantly improves performance in material property prediction and human-AI interaction, surpassing general-purpose LLMs such as GPT-4. We also demonstrate its usefulness in applications such as more advanced scientific reasoning and step-by-step material synthesis.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

22 of 45 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023Mistral 7Bcited by 3,164
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
10 more
  1. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 571
  2. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  3. 2021Crystal Diffusion Variational Autoencoder for Periodic Material Generationcited by 270
  4. 2023MatterGen: a generative model for inorganic materials designcited by 98
  5. 2019DScribe: Library of Descriptors for Machine Learning in Materials Sciencecited by 63
  6. 2022Structured information extraction from complex scientific text with fine-tuned large language modelscited by 56
  7. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  8. 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMscited by 24
  9. 2024Symbolic Working Memory Enhances Language Models for Complex Rule Applicationcited by 17
  10. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 7.

18 papers.

Sort:

By area:

  1. 2025OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discoverycited by 20
  2. 2025MooseAgent: A LLM Based Multi-agent Framework for Automating Moose Simulationcited by 17
  3. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterizationcited by 9
  4. 2025A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Toolscited by 7
  5. 2026PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVRcited by 6
  6. 2025AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materialscited by 5
  7. 2025Stress-Testing Multimodal Foundation Models for Crystallographic Reasoningcited by 4
  8. 2026Towards Agentic Intelligence for Materials Sciencecited by 3
  9. 2025Uncovering Bottlenecks and Optimizing Scientific Lab Workflows with Cycle Time Reduction Agentscited by 2
  10. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  11. 2026AutoSAM: an Agentic Framework for Automating Input File Generation for the SAM Code with Multi-Modal Retrieval-Augmented Generationcited by 2
  12. 2025Perovskite-R1: a domain-specialized large language model for intelligent discovery of precursor additives and experimental designcited by 1
6 more
  1. 2025Role of Large Language Models and Retrieval-Augmented Generation for Accelerating Crystalline Material Discovery: A Systematic Reviewcited by 0
  2. 2025Language-Native Materials Processing Design by Lightly Structured Text Database and Reasoning Large Language Modelcited by 0
  3. 2026Scaling-Aware Adapter for Structure-Grounded LLM Reasoningcited by 0
  4. 2026SciCore-Mol: Augmenting Large Language Models with Pluggable Molecular Cognition Modulescited by 0
  5. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  6. 2026Physics as the label for measuring and correcting materials reasoning in multimodal modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Foundational Large Language Models for Materials Research
  2. 2025Large language models in materials science and the need for open-source approaches
  3. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation
  4. 2025MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis
  5. 2026From Data to Theory: Autonomous Large Language Model Agents for Materials Science
  6. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
  7. 2024Are LLMs Ready for Real-World Materials Discovery?
  8. 2026MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models