# Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders

arXiv: https://arxiv.org/abs/1806.09300 · HTML: https://mlsanity.com/paper/1806.09300/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2018.

## Built on by (most cited first, 11 of 11)

- [MGCVAE: Multi-objective Inverse Design via Molecular Graph Conditional Variational Autoencoder](https://mlsanity.com/paper/2202.07476/) (2022)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [Difficulty in chirality recognition for Transformer architectures learning chemical structures from string](https://mlsanity.com/paper/2303.11593/) (2023)
- [Graph Machine Learning for Design of High-Octane Fuels](https://mlsanity.com/paper/2206.00619/) (2022)
- [Faster and more diverse de novo molecular optimization with double-loop reinforcement learning using augmented SMILES](https://mlsanity.com/paper/2210.12458/) (2022)
- [MolX: Enhancing Large Language Models for Molecular Understanding With A Multi-Modal Extension](https://mlsanity.com/paper/2406.06777/) (2024)
- [A novel molecule generative model of VAE combined with Transformer for unseen structure generation](https://mlsanity.com/paper/2402.11950/) (2024)
- [An Unpooling Layer for Graph Generation](https://mlsanity.com/paper/2206.01874/) (2022)
- [CRNNTL: convolutional recurrent neural network and transfer learning for QSAR modelling](https://mlsanity.com/paper/2109.03309/) (2021)
- [A Generative Model for Disentangling Galaxy Photometric Parameters](https://mlsanity.com/paper/2507.15898/) (2025)
- [Molecular Representations for AI in Chemistry and Materials Science: An NLP Perspective](https://mlsanity.com/paper/2603.05525/) (2026)

## Abstract (arXiv)

Chemical autoencoders are attractive models as they combine chemical space navigation with possibilities for de-novo molecule generation in areas of interest. This enables them to produce focused chemical libraries around a single lead compound for employment early in a drug discovery project. Here it is shown that the choice of chemical representation, such as SMILES strings, has a large influence on the properties of the latent space. It is further explored to what extent translating between different chemical representations influences the latent space similarity to the SMILES strings or circular fingerprints. By employing SMILES enumeration for either the encoder or decoder, it is found that the decoder has the largest influence on the properties of the latent space. Training a sequence to sequence heteroencoder based on recurrent neural networks(RNNs) with long short-term memory cells (LSTM) to predict different enumerated SMILES strings from the same canonical SMILES string gives the largest similarity between latent space distance and molecular similarity measured as circular fingerprints similarity. Using the output from the bottleneck in QSAR modelling of five molecular datasets shows that heteroencoder derived vectors markedly outperforms autoencoder derived vectors as well as models built using ECFP4 fingerprints, underlining the increased chemical relevance of the latent space. However, the use of enumeration during training of the decoder leads to a markedly increase in the rate of decoding to a different molecules than encoded, a tendency that can be counteracted with more complex network architectures.