mlsanityData to 7 Oct 2026

Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

arXiv 2402.04379 ↗First posted 6 Feb 2024cs.LG cond-mat.mtrl-scicited by 92

Not on the map. Why?

Abstract (arXiv)

We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

26 of 42 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  9. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  10. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  12. 2021Crystal Diffusion Variational Autoencoder for Periodic Material Generationcited by 268
14 more
  1. 2023Language Modeling Is Compressioncited by 225
  2. 2023Large Language Models as General Pattern Machinescited by 196
  3. 2023Protein Design with Guided Discrete Diffusioncited by 154
  4. 2022Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoderscited by 99
  5. 2023MatterGen: a generative model for inorganic materials designcited by 98
  6. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  7. 2023Stay on topic with Classifier-Free Guidancecited by 90
  8. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  9. 2023Large Language Models can Learn Rulescited by 66
  10. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  11. 2023Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB filescited by 59
  12. 2023Scalable Diffusion for Materials Generationcited by 52
  13. 2022The Lie Derivative for Measuring Learned Equivariancecited by 51
  14. 2023Atom-by-atom protein generation and beyond with language modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 48; 2026: 23.

92 papers.

Sort:

By area:

  1. 2024FlowMM: Generating Materials with Riemannian Flow Matchingcited by 105
  2. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  3. 2025All-atom Diffusion Transformers: Unified generative modelling of molecules and materialscited by 75
  4. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 54
  5. 2024FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributionscited by 52
  6. 2025SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Modelscited by 48
  7. 2024LLMatDesign: Autonomous Materials Discovery with Large Language Modelscited by 41
  8. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  9. 2024Space Group Informed Transformer for Crystalline Materials Generationcited by 40
  10. 2024Are large language models superhuman chemists?cited by 34
  11. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  12. 2024FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Modelscited by 33
38 more
  1. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  2. 2025Open Materials Generation with Stochastic Interpolantscited by 26
  3. 2025Wyckoff Transformer: Generation of Symmetric Crystalscited by 24
  4. 2025A practical guide to machine learning interatomic potentials -- Status and futurecited by 24
  5. 2024Materials science in the era of large language models: a perspectivecited by 23
  6. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understandingcited by 23
  7. 2024LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Predictioncited by 20
  8. 2025LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Modelscited by 20
  9. 2025CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editingcited by 19
  10. 2025Guiding Generative Models to Uncover Diverse and Novel Crystals via Reinforcement Learningcited by 19
  11. 2024MatExpert: Decomposing Materials Discovery by Mimicking Human Expertscited by 18
  12. 2024Foundational Large Language Models for Materials Researchcited by 18
  13. 2025A Periodic Bayesian Flow for Material Generationcited by 15
  14. 2025Periodic Materials Generation using Text-Guided Joint Diffusion Modelcited by 15
  15. 2025Space Group Equivariant Crystal Diffusioncited by 15
  16. 2025MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Sciencecited by 15
  17. 2024Generative Hierarchical Materials Searchcited by 12
  18. 2025Invariant Tokenization of Crystalline Materials for Language Model Enabled Generationcited by 12
  19. 2025LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generationcited by 12
  20. 2025Reinforcement Fine-Tuning for Materials Designcited by 11
  21. 2025PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Designcited by 11
  22. 2024Less can be more for predicting properties with large language modelscited by 10
  23. 2025Large Language-Geometry Model: When LLM meets Equivariancecited by 10
  24. 2025Toward Greater Autonomy in Materials Discovery Agents: Unifying Planning, Physics, and Scientistscited by 10
  25. 2024Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversitycited by 9
  26. 2025Symmetry-Aware Bayesian Flow Networks for Crystal Generationcited by 9
  27. 2025Materials Generation in the Era of Artificial Intelligence: A Comprehensive Surveycited by 9
  28. 2025Kinetic Langevin Diffusion for Crystalline Materials Generationcited by 9
  29. 2026Crystalite: A Lightweight Transformer for Efficient Crystal Modelingcited by 9
  30. 2024MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Designcited by 8
  31. 2025Establishing baselines for generative discovery of inorganic crystalscited by 8
  32. 2025MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Modelscited by 7
  33. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 7
  34. 2025Scalable Autoregressive 3D Molecule Generationcited by 7
  35. 2025A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Toolscited by 7
  36. 2025CrystalGRW: Generative Modeling of Crystal Structures with Targeted Properties via Geodesic Random Walkscited by 5
  37. 2025deCIFer: Crystal Structure Prediction from Powder Diffraction Data using Autoregressive Language Modelscited by 5
  38. 2025AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materialscited by 5

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Atomistic Language Models Understand and Generate Materials
  2. 2025MatterChat: A Multi-Modal LLM for Material Science
  3. 2025Large language models in materials science and the need for open-source approaches
  4. 2026From Data to Theory: Autonomous Large Language Model Agents for Materials Science
  5. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation
  6. 2026MatMind: A Structure-Activity Knowledge-Driven Generative Foundation Model for Materials Science
  7. 2026SCALAR: Quantifying Structural Hallucination, Consistency, and Reasoning Gaps in Materials Foundation Models
  8. 2025Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis