# LLamol: A Dynamic Multi-Conditional Generative Transformer for De Novo Molecular Design

arXiv: https://arxiv.org/abs/2311.14407 · HTML: https://mlsanity.com/paper/2311.14407/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [GuacaMol: Benchmarking Models for De Novo Molecular Design](https://mlsanity.com/paper/1811.09621/) (2018)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Molecular generative model based on conditional variational autoencoder for de novo molecular design](https://mlsanity.com/paper/1806.05805/) (2018)
- [MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design](https://mlsanity.com/paper/2203.14500/) (2022)
- [Message-passing neural networks for high-throughput polymer screening](https://mlsanity.com/paper/1807.10363/) (2018)
- [Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention](https://mlsanity.com/paper/2103.00213/) (2021)
- [MGCVAE: Multi-objective Inverse Design via Molecular Graph Conditional Variational Autoencoder](https://mlsanity.com/paper/2202.07476/) (2022)
- [Conditional β-VAE for De Novo Molecular Generation](https://mlsanity.com/paper/2205.01592/) (2022)

## Built on by (most cited first, 6 of 6)

- [Improved Molecular Generation through Attribute-Driven Integrative Embeddings and GAN Selectivity](https://mlsanity.com/paper/2504.19040/) (2025)
- [Conditional Chemical Language Models are Versatile Tools in Drug Discovery](https://mlsanity.com/paper/2507.10273/) (2025)
- [A Transformer Model for Predicting Chemical Products from Generic SMARTS Templates with Data Augmentation](https://mlsanity.com/paper/2503.05810/) (2025)
- [ChemFixer: Correcting Invalid Molecules to Unlock Previously Unseen Chemical Space](https://mlsanity.com/paper/2511.13758/) (2025)
- [Materium: An Autoregressive Approach for Material Generation](https://mlsanity.com/paper/2512.07486/) (2025)
- [Flexible Cutoff Learning: Optimizing Machine Learning Potentials After Training](https://mlsanity.com/paper/2603.10205/) (2026)

## Abstract (arXiv)

Generative models have demonstrated substantial promise in Natural Language Processing (NLP) and have found application in designing molecules, as seen in General Pretrained Transformer (GPT) models. In our efforts to develop such a tool for exploring the organic chemical space in search of potentially electro-active compounds, we present "LLamol", a single novel generative transformer model based on the LLama 2 architecture, which was trained on a 13M superset of organic compounds drawn from diverse public sources. To allow for a maximum flexibility in usage and robustness in view of potentially incomplete data, we introduce "Stochastic Context Learning" as a new training procedure. We demonstrate that the resulting model adeptly handles single- and multi-conditional organic molecule generation with up to four conditions, yet more are possible. The model generates valid molecular structures in SMILES notation while flexibly incorporating three numerical and/or one token sequence into the generative process, just as requested. The generated compounds are very satisfactory in all scenarios tested. In detail, we showcase the model's capability to utilize token sequences for conditioning, either individually or in combination with numerical properties, making LLamol a potent tool for de novo molecule design, easily expandable with new properties.