# FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions

arXiv: https://arxiv.org/abs/2410.23405 · HTML: https://mlsanity.com/paper/2410.23405/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Flow matching, time series and science, one of 14 directions in Diffusion models.
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Placed in

- Diffusion models / [Flow matching, time series and science](https://mlsanity.com/diffusion/flow-matching-time-series-and-science/)

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities](https://mlsanity.com/paper/2006.02425/) (2020)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [e3nn: Euclidean Neural Networks](https://mlsanity.com/paper/2207.09453/) (2022)
- [Crystal Structure Prediction by Joint Equivariant Diffusion](https://mlsanity.com/paper/2309.04475/) (2023)
- [Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs](https://mlsanity.com/paper/2302.03655/) (2023)
- [FlowMM: Generating Materials with Riemannian Flow Matching](https://mlsanity.com/paper/2406.04713/) (2024)
- [MatterGen: a generative model for inorganic materials design](https://mlsanity.com/paper/2312.03687/) (2023)
- [Fine-Tuned Language Models Generate Stable Inorganic Materials as Text](https://mlsanity.com/paper/2402.04379/) (2024)
- [Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks](https://mlsanity.com/paper/2305.14201/) (2023)
- [Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds](https://mlsanity.com/paper/2106.06020/) (2021)
- [Space Group Constrained Crystal Generation](https://mlsanity.com/paper/2402.03992/) (2024)
- [Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files](https://mlsanity.com/paper/2305.05708/) (2023)
- [Scalable Diffusion for Materials Generation](https://mlsanity.com/paper/2311.09235/) (2023)
- [An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties](https://mlsanity.com/paper/2005.07609/) (2020)
- [Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties](https://mlsanity.com/paper/2008.08461/) (2020)
- [Spherical Channels for Modeling Atomic Interactions](https://mlsanity.com/paper/2206.14331/) (2022)
- [Normalizing flows for atomic solids](https://mlsanity.com/paper/2111.08696/) (2021)
- [The Open DAC 2023 Dataset and Challenges for Sorbent Discovery in Direct Air Capture](https://mlsanity.com/paper/2311.00341/) (2023)
- [Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling](https://mlsanity.com/paper/2308.02165/) (2023)

## Built on by (most cited first, 30 of 53)

- [All-atom Diffusion Transformers: Unified generative modelling of molecules and materials](https://mlsanity.com/paper/2503.03965/) (2025)
- [Open Materials Generation with Stochastic Interpolants](https://mlsanity.com/paper/2502.02582/) (2025)
- [LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models](https://mlsanity.com/paper/2512.04562/) (2025)
- [Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces](https://mlsanity.com/paper/2506.07903/) (2025)
- [Space Group Equivariant Crystal Diffusion](https://mlsanity.com/paper/2505.10994/) (2025)
- [PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Design](https://mlsanity.com/paper/2509.07150/) (2025)
- [LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation](https://mlsanity.com/paper/2510.23040/) (2025)
- [All that structure matches does not glitter](https://mlsanity.com/paper/2509.12178/) (2025)
- [AI-driven inverse design of materials: Past, present and future](https://mlsanity.com/paper/2411.09429/) (2024)
- [Materials Generation in the Era of Artificial Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2505.16379/) (2025)
- [Toward Greater Autonomy in Materials Discovery Agents: Unifying Planning, Physics, and Scientists](https://mlsanity.com/paper/2506.05616/) (2025)
- [Crystalite: A Lightweight Transformer for Efficient Crystal Modeling](https://mlsanity.com/paper/2604.02270/) (2026)
- [Establishing baselines for generative discovery of inorganic crystals](https://mlsanity.com/paper/2501.02144/) (2025)
- [Symmetry-Aware Bayesian Flow Networks for Crystal Generation](https://mlsanity.com/paper/2502.03146/) (2025)
- [Kinetic Langevin Diffusion for Crystalline Materials Generation](https://mlsanity.com/paper/2507.03602/) (2025)
- [Artificial Intelligence and Generative Models for Materials Discovery -- A Review](https://mlsanity.com/paper/2508.03278/) (2025)
- [Open Materials Generation with Inference-Time Reinforcement Learning](https://mlsanity.com/paper/2602.00424/) (2026)
- [MatLLMSearch: Crystal Structure Discovery with Evolution-Guided Large Language Models](https://mlsanity.com/paper/2502.20933/) (2025)
- [Language Models Enable Data-Augmented Synthesis Planning for Inorganic Materials](https://mlsanity.com/paper/2506.12557/) (2025)
- [A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools](https://mlsanity.com/paper/2506.20743/) (2025)
- [Space Group Conditional Flow Matching](https://mlsanity.com/paper/2509.23822/) (2025)
- [Multimodal Crystal Flow: Any-to-Any Modality Generation for Unified Crystal Modeling](https://mlsanity.com/paper/2602.20210/) (2026)
- [Efficient Flow Matching using Latent Variables](https://mlsanity.com/paper/2505.04486/) (2025)
- [TopoMAS: Large Language Model Driven Topological Materials Multiagent System](https://mlsanity.com/paper/2507.04053/) (2025)
- [CrystalDiT: A Diffusion Transformer for Crystal Generation](https://mlsanity.com/paper/2508.16614/) (2025)
- [Continuous SUN (Stable, Unique, and Novel) Metric for Generative Modeling of Inorganic Crystals](https://mlsanity.com/paper/2510.12405/) (2025)
- [Boltzmann generators for amorphous particle systems](https://mlsanity.com/paper/2512.16607/) (2025)
- [CrystalGRW: Generative Modeling of Crystal Structures with Targeted Properties via Geodesic Random Walks](https://mlsanity.com/paper/2501.08998/) (2025)
- [Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materials](https://mlsanity.com/paper/2602.22251/) (2026)
- [MiAD: Mirage Atom Diffusion for De Novo Crystal Generation](https://mlsanity.com/paper/2511.14426/) (2025)

## Abstract (arXiv)

Material discovery is a critical area of research with the potential to revolutionize various fields, including carbon capture, renewable energy, and electronics. However, the immense scale of the chemical space makes it challenging to explore all possible materials experimentally. In this paper, we introduce FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. FlowLLM first fine-tunes an LLM to learn an effective base distribution of meta-stable crystals in a text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. Our approach significantly outperforms state-of-the-art methods, increasing the generation rate of stable materials by over three times and increasing the rate for stable, unique, and novel crystals by ~50% - a huge improvement on a difficult problem. Additionally, the crystals generated by FlowLLM are much closer to their relaxed state when compared with another leading model, significantly reducing post-hoc computational cost.