# A novel molecule generative model of VAE combined with Transformer for unseen structure generation

arXiv: https://arxiv.org/abs/2402.11950 · HTML: https://mlsanity.com/paper/2402.11950/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Grammar Variational Autoencoder](https://mlsanity.com/paper/1703.01925/) (2017)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Constrained Graph Variational Autoencoders for Molecule Design](https://mlsanity.com/paper/1805.09076/) (2018)
- [Molecular generative model based on conditional variational autoencoder for de novo molecular design](https://mlsanity.com/paper/1806.05805/) (2018)
- [Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention](https://mlsanity.com/paper/2103.00213/) (2021)
- [Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders](https://mlsanity.com/paper/1806.09300/) (2018)
- [Difficulty in chirality recognition for Transformer architectures learning chemical structures from string](https://mlsanity.com/paper/2303.11593/) (2023)
- [Investigation of a Data Split Strategy Involving the Time Axis in Adverse Event Prediction Using Machine Learning](https://mlsanity.com/paper/2204.08682/) (2022)

## Built on by (most cited first, 4 of 4)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [ReACT-Drug: Reaction-Template Guided Reinforcement Learning for de novo Drug Design](https://mlsanity.com/paper/2512.20958/) (2025)
- [From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models](https://mlsanity.com/paper/2605.09949/) (2026)
- [A Systematic Evaluation of Molecule Generation Models for De Novo Drug Design: From Benchmarks to Practical Insights](https://mlsanity.com/paper/2609.10099/) (2026)

## Abstract (arXiv)

Recently, molecule generation using deep learning has been actively investigated in drug discovery. In this field, Transformer and VAE are widely used as powerful models, but they are rarely used in combination due to structural and performance mismatch of them. This study proposes a model that combines these two models through structural and parameter optimization in handling diverse molecules. The proposed model shows comparable performance to existing models in generating molecules, and showed by far superior performance in generating molecules with unseen structures. Another advantage of this VAE model is that it generates molecules from latent representation, and therefore properties of molecules can be easily predicted or conditioned with it, and indeed, we show that the latent representation of the model successfully predicts molecular properties. Ablation study suggested the advantage of VAE over other generative models like language model in generating novel molecules. It also indicated that the latent representation can be shortened to ~32 dimensional variables without loss of reconstruction, suggesting the possibility of a much smaller molecular descriptor or model than existing ones. This study is expected to provide a virtual chemical library containing a wide variety of compounds for virtual screening and to enable efficient screening.