# Keeping it Simple: Language Models can learn Complex Molecular Distributions

arXiv: https://arxiv.org/abs/2112.03041 · HTML: https://mlsanity.com/paper/2112.03041/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 69 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Efficient Graph Generation with Graph Recurrent Attention Networks](https://mlsanity.com/paper/1910.00760/) (2019)
- [Learning Deep Generative Models of Graphs](https://mlsanity.com/paper/1803.03324/) (2018)
- [Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules](https://mlsanity.com/paper/1906.00957/) (2019)
- [GraphNVP: An Invertible Flow Model for Generating Molecular Graphs](https://mlsanity.com/paper/1905.11600/) (2019)
- [Syntax-Directed Variational Autoencoder for Structured Data](https://mlsanity.com/paper/1802.08786/) (2018)
- [Multi-Objective De Novo Drug Design with Conditional Graph Generative Model](https://mlsanity.com/paper/1801.07299/) (2018)
- [MolecularRNN: Generating realistic molecular graphs with optimized properties](https://mlsanity.com/paper/1905.13372/) (2019)
- [NeVAE: A Deep Generative Model for Molecular Graphs](https://mlsanity.com/paper/1802.05283/) (2018)
- [Categorical Normalizing Flows via Continuous Transformations](https://mlsanity.com/paper/2006.09790/) (2020)
- [Message-passing neural networks for high-throughput polymer screening](https://mlsanity.com/paper/1807.10363/) (2018)
- [Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation](https://mlsanity.com/paper/1905.13741/) (2019)
- [Neural Message Passing on High Order Paths](https://mlsanity.com/paper/2002.10413/) (2020)
- [Graph Deconvolutional Generation](https://mlsanity.com/paper/2002.07087/) (2020)

## Built on by (most cited first, 30 of 68)

- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)
- [Equivariant Energy-Guided SDE for Inverse Molecular Design](https://mlsanity.com/paper/2209.15408/) (2022)
- [MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design](https://mlsanity.com/paper/2203.14500/) (2022)
- [Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files](https://mlsanity.com/paper/2305.05708/) (2023)
- [Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo](https://mlsanity.com/paper/2504.13139/) (2025)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast](https://mlsanity.com/paper/2202.09346/) (2022)
- [Group SELFIES: A Robust Fragment-Based Molecular String Representation](https://mlsanity.com/paper/2211.13322/) (2022)
- [Computer-Aided Multi-Objective Optimization in Small Molecule Discovery](https://mlsanity.com/paper/2210.07209/) (2022)
- [nach0: Multimodal Natural and Chemical Languages Foundation Model](https://mlsanity.com/paper/2311.12410/) (2023)
- [Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trials](https://mlsanity.com/paper/2409.04481/) (2024)
- [Domain-Agnostic Molecular Generation with Chemical Feedback](https://mlsanity.com/paper/2301.11259/) (2023)
- [Forecasting Rare Language Model Behaviors](https://mlsanity.com/paper/2502.16797/) (2025)
- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation](https://mlsanity.com/paper/2502.12638/) (2025)
- [An Interdisciplinary Outlook on Large Language Models for Scientific Research](https://mlsanity.com/paper/2311.04929/) (2023)
- [Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation](https://mlsanity.com/paper/2405.17066/) (2024)
- [t-SMILES: A Scalable Fragment-based Molecular Representation Framework for De Novo Molecule Generation](https://mlsanity.com/paper/2301.01829/) (2023)
- [SmileyLlama: Modifying Large Language Models for Directed Chemical Space Exploration](https://mlsanity.com/paper/2409.02231/) (2024)
- [Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences](https://mlsanity.com/paper/2411.04165/) (2024)
- [BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning](https://mlsanity.com/paper/2406.03686/) (2024)
- [Generative Adversarial Model-Based Optimization via Source Critic Regularization](https://mlsanity.com/paper/2402.06532/) (2024)
- [SynLlama: Generating Synthesizable Molecules and Their Analogs with Large Language Models](https://mlsanity.com/paper/2503.12602/) (2025)
- [Probabilistic Generative Transformer Language models for Generative Design of Molecules](https://mlsanity.com/paper/2209.09406/) (2022)
- [Augmented Memory: Capitalizing on Experience Replay to Accelerate De Novo Molecular Design](https://mlsanity.com/paper/2305.16160/) (2023)
- [3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation](https://mlsanity.com/paper/2403.07179/) (2024)
- [Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models](https://mlsanity.com/paper/2412.13791/) (2024)

## Abstract (arXiv)

Deep generative models of molecules have grown immensely in popularity, trained on relevant datasets, these models are used to search through chemical space. The downstream utility of generative models for the inverse design of novel functional compounds depends on their ability to learn a training distribution of molecules. The most simple example is a language model that takes the form of a recurrent neural network and generates molecules using a string representation. More sophisticated are graph generative models, which sequentially construct molecular graphs and typically achieve state of the art results. However, recent work has shown that language models are more capable than once thought, particularly in the low data regime. In this work, we investigate the capacity of simple language models to learn distributions of molecules. For this purpose, we introduce several challenging generative modeling tasks by compiling especially complex distributions of molecules. On each task, we evaluate the ability of language models as compared with two widely used graph generative models. The results demonstrate that language models are powerful generative models, capable of adeptly learning complex molecular distributions -- and yield better performance than the graph models. Language models can accurately generate: distributions of the highest scoring penalized LogP molecules in ZINC15, multi-modal molecular distributions as well as the largest molecules in PubChem.