# Faster and more diverse de novo molecular optimization with double-loop reinforcement learning using augmented SMILES

arXiv: https://arxiv.org/abs/2210.12458 · HTML: https://mlsanity.com/paper/2210.12458/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 46 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [GuacaMol: Benchmarking Models for De Novo Molecular Design](https://mlsanity.com/paper/1811.09621/) (2018)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization](https://mlsanity.com/paper/2206.12411/) (2022)
- [SMILES Enumeration as Data Augmentation for Neural Network Modeling of Molecules](https://mlsanity.com/paper/1703.07076/) (2017)
- [Deep learning for molecular design - a review of the state of the art](https://mlsanity.com/paper/1903.04388/) (2019)
- [Molecular Generation with Recurrent Neural Networks (RNNs)](https://mlsanity.com/paper/1705.04612/) (2017)
- [Generating Focussed Molecule Libraries for Drug Discovery with Recurrent Neural Networks](https://mlsanity.com/paper/1701.01329/) (2017)
- [Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders](https://mlsanity.com/paper/1806.09300/) (2018)
- [Conditional β-VAE for De Novo Molecular Generation](https://mlsanity.com/paper/2205.01592/) (2022)

## Built on by (most cited first, 5 of 5)

- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation](https://mlsanity.com/paper/2405.17066/) (2024)
- [Augmented Memory: Capitalizing on Experience Replay to Accelerate De Novo Molecular Design](https://mlsanity.com/paper/2305.16160/) (2023)
- [Test-Time Training Scaling Laws for Chemical Exploration in Drug Design](https://mlsanity.com/paper/2501.19153/) (2025)
- [Evaluating Effects of Augmented SELFIES for Molecular Understanding Using QK-LSTM](https://mlsanity.com/paper/2504.20789/) (2025)

## Abstract (arXiv)

Using generative deep learning models and reinforcement learning together can effectively generate new molecules with desired properties. By employing a multi-objective scoring function, thousands of high-scoring molecules can be generated, making this approach useful for drug discovery and material science. However, the application of these methods can be hindered by computationally expensive or time-consuming scoring procedures, particularly when a large number of function calls are required as feedback in the reinforcement learning optimization. Here, we propose the use of double-loop reinforcement learning with simplified molecular line entry system (SMILES) augmentation to improve the efficiency and speed of the optimization. By adding an inner loop that augments the generated SMILES strings to non-canonical SMILES for use in additional reinforcement learning rounds, we can both reuse the scoring calculations on the molecular level, thereby speeding up the learning process, as well as offer additional protection against mode collapse. We find that employing between 5 and 10 augmentation repetitions is optimal for the scoring functions tested and is further associated with an increased diversity in the generated compounds, improved reproducibility of the sampling runs and the generation of molecules of higher similarity to known ligands.