# Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files

arXiv: https://arxiv.org/abs/2305.05708 · HTML: https://mlsanity.com/paper/2305.05708/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 47 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Learning Deep Generative Models of Graphs](https://mlsanity.com/paper/1803.03324/) (2018)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules](https://mlsanity.com/paper/1906.00957/) (2019)
- [E(n) Equivariant Normalizing Flows](https://mlsanity.com/paper/2105.09016/) (2021)
- [A 3D Generative Model for Structure-Based Drug Design](https://mlsanity.com/paper/2203.10446/) (2022)
- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)
- [Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2110.04624/) (2021)
- [Keeping it Simple: Language Models can learn Complex Molecular Distributions](https://mlsanity.com/paper/2112.03041/) (2021)
- [Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation](https://mlsanity.com/paper/1905.13741/) (2019)
- [Neural Message Passing on High Order Paths](https://mlsanity.com/paper/2002.10413/) (2020)

## Built on by (most cited first, 30 of 59)

- [FlowMM: Generating Materials with Riemannian Flow Matching](https://mlsanity.com/paper/2406.04713/) (2024)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Fine-Tuned Language Models Generate Stable Inorganic Materials as Text](https://mlsanity.com/paper/2402.04379/) (2024)
- [14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon](https://mlsanity.com/paper/2306.06283/) (2023)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [All-atom Diffusion Transformers: Unified generative modelling of molecules and materials](https://mlsanity.com/paper/2503.03965/) (2025)
- [Efficient Evolutionary Search Over Chemical Space with Large Language Models](https://mlsanity.com/paper/2406.16976/) (2024)
- [Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective](https://mlsanity.com/paper/2412.03487/) (2024)
- [Swallowing the Bitter Pill: Simplified Scalable Conformer Generation](https://mlsanity.com/paper/2311.17932/) (2023)
- [Scalable Diffusion for Materials Generation](https://mlsanity.com/paper/2311.09235/) (2023)
- [FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions](https://mlsanity.com/paper/2410.23405/) (2024)
- [SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models](https://mlsanity.com/paper/2502.03638/) (2025)
- [LLMatDesign: Autonomous Materials Discovery with Large Language Models](https://mlsanity.com/paper/2406.13163/) (2024)
- [Space Group Informed Transformer for Crystalline Materials Generation](https://mlsanity.com/paper/2403.15734/) (2024)
- [3D molecule generation by denoising voxel grids](https://mlsanity.com/paper/2306.07473/) (2023)
- [Are LLMs Ready for Real-World Materials Discovery?](https://mlsanity.com/paper/2402.05200/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Are large language models superhuman chemists?](https://mlsanity.com/paper/2404.01475/) (2024)
- [LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models](https://mlsanity.com/paper/2512.04562/) (2025)
- [Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials](https://mlsanity.com/paper/2306.09375/) (2023)
- [LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction](https://mlsanity.com/paper/2411.00177/) (2024)
- [Structure-based drug design by denoising voxel grids](https://mlsanity.com/paper/2405.03961/) (2024)
- [MatExpert: Decomposing Materials Discovery by Mimicking Human Experts](https://mlsanity.com/paper/2410.21317/) (2024)
- [Transformers and Large Language Models for Chemistry and Drug Discovery](https://mlsanity.com/paper/2310.06083/) (2023)
- [Generative Hierarchical Materials Search](https://mlsanity.com/paper/2409.06762/) (2024)
- [Reinforcement Fine-Tuning for Materials Design](https://mlsanity.com/paper/2504.02367/) (2025)
- [BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning](https://mlsanity.com/paper/2406.03686/) (2024)
- [Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation](https://mlsanity.com/paper/2503.00152/) (2025)
- [Less can be more for predicting properties with large language models](https://mlsanity.com/paper/2406.17295/) (2024)

## Abstract (arXiv)

Language models are powerful tools for molecular design. Currently, the dominant paradigm is to parse molecular graphs into linear string representations that can easily be trained on. This approach has been very successful, however, it is limited to chemical structures that can be completely represented by a graph -- like organic molecules -- while materials and biomolecular structures like protein binding sites require a more complete representation that includes the relative positioning of their atoms in space. In this work, we show how language models, without any architecture modifications, trained using next-token prediction -- can generate novel and valid structures in three dimensions from various substantially different distributions of chemical structures. In particular, we demonstrate that language models trained directly on sequences derived directly from chemical file formats like XYZ files, Crystallographic Information files (CIFs), or Protein Data Bank files (PDBs) can directly generate molecules, crystals, and protein binding sites in three dimensions. Furthermore, despite being trained on chemical file sequences -- language models still achieve performance comparable to state-of-the-art models that use graph and graph-derived string representations, as well as other domain-specific 3D generative models. In doing so, we demonstrate that it is not necessary to use simplified molecular representations to train chemical language models -- that they are powerful generative models capable of directly exploring chemical space in three dimensions for very different structures.