# Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

arXiv: https://arxiv.org/abs/2402.04379 · HTML: https://mlsanity.com/paper/2402.04379/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders](https://mlsanity.com/paper/2203.12742/) (2022)
- [MatterGen: a generative model for inorganic materials design](https://mlsanity.com/paper/2312.03687/) (2023)
- [AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model](https://mlsanity.com/paper/2309.16058/) (2023)
- [Stay on topic with Classifier-Free Guidance](https://mlsanity.com/paper/2306.17806/) (2023)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [Large Language Models can Learn Rules](https://mlsanity.com/paper/2310.07064/) (2023)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files](https://mlsanity.com/paper/2305.05708/) (2023)
- [The Lie Derivative for Measuring Learned Equivariance](https://mlsanity.com/paper/2210.02984/) (2022)
- [Scalable Diffusion for Materials Generation](https://mlsanity.com/paper/2311.09235/) (2023)
- [Atom-by-atom protein generation and beyond with language models](https://mlsanity.com/paper/2308.09482/) (2023)

## Built on by (most cited first, 30 of 93)

- [FlowMM: Generating Materials with Riemannian Flow Matching](https://mlsanity.com/paper/2406.04713/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [All-atom Diffusion Transformers: Unified generative modelling of molecules and materials](https://mlsanity.com/paper/2503.03965/) (2025)
- [Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective](https://mlsanity.com/paper/2412.03487/) (2024)
- [FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions](https://mlsanity.com/paper/2410.23405/) (2024)
- [SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models](https://mlsanity.com/paper/2502.03638/) (2025)
- [LLMatDesign: Autonomous Materials Discovery with Large Language Models](https://mlsanity.com/paper/2406.13163/) (2024)
- [Space Group Informed Transformer for Crystalline Materials Generation](https://mlsanity.com/paper/2403.15734/) (2024)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Are large language models superhuman chemists?](https://mlsanity.com/paper/2404.01475/) (2024)
- [FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Models](https://mlsanity.com/paper/2411.05823/) (2024)
- [Probing the limitations of multimodal language models for chemistry and materials research](https://mlsanity.com/paper/2411.16955/) (2024)
- [Open Materials Generation with Stochastic Interpolants](https://mlsanity.com/paper/2502.02582/) (2025)
- [Wyckoff Transformer: Generation of Symmetric Crystals](https://mlsanity.com/paper/2503.02407/) (2025)
- [A practical guide to machine learning interatomic potentials -- Status and future](https://mlsanity.com/paper/2503.09814/) (2025)
- [Materials science in the era of large language models: a perspective](https://mlsanity.com/paper/2403.06949/) (2024)
- [MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding](https://mlsanity.com/paper/2407.04903/) (2024)
- [LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models](https://mlsanity.com/paper/2512.04562/) (2025)
- [LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction](https://mlsanity.com/paper/2411.00177/) (2024)
- [Guiding Generative Models to Uncover Diverse and Novel Crystals via Reinforcement Learning](https://mlsanity.com/paper/2511.07158/) (2025)
- [CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editing](https://mlsanity.com/paper/2502.03997/) (2025)
- [MatExpert: Decomposing Materials Discovery by Mimicking Human Experts](https://mlsanity.com/paper/2410.21317/) (2024)
- [Foundational Large Language Models for Materials Research](https://mlsanity.com/paper/2412.09560/) (2024)
- [A Periodic Bayesian Flow for Material Generation](https://mlsanity.com/paper/2502.02016/) (2025)
- [Periodic Materials Generation using Text-Guided Joint Diffusion Model](https://mlsanity.com/paper/2503.00522/) (2025)
- [Space Group Equivariant Crystal Diffusion](https://mlsanity.com/paper/2505.10994/) (2025)
- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Design](https://mlsanity.com/paper/2509.07150/) (2025)
- [Generative Hierarchical Materials Search](https://mlsanity.com/paper/2409.06762/) (2024)

## Abstract (arXiv)

We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.