# Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention

arXiv: https://arxiv.org/abs/2103.00213 · HTML: https://mlsanity.com/paper/2103.00213/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 27 Feb 2021.

## Builds on (most cited first, 17 of 17)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discovery](https://mlsanity.com/paper/1803.09518/) (2018)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Learning to Plan Chemical Syntheses](https://mlsanity.com/paper/1708.04202/) (2017)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space](https://mlsanity.com/paper/1909.11655/) (2019)
- [ChemTS: An Efficient Python Library for de novo Molecular Generation](https://mlsanity.com/paper/1710.00616/) (2017)
- [Application of generative autoencoder in de novo molecular design](https://mlsanity.com/paper/1711.07839/) (2017)

## Built on by (most cited first, 20 of 20)

- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)
- [Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents](https://mlsanity.com/paper/2410.13185/) (2024)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [A smile is all you need: Predicting limiting activity coefficients from SMILES with natural language processing](https://mlsanity.com/paper/2206.07048/) (2022)
- [SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks](https://mlsanity.com/paper/2410.03769/) (2024)
- [SPT-NRTL: A physics-guided machine learning model to predict thermodynamically consistent activity coefficients](https://mlsanity.com/paper/2209.04135/) (2022)
- [Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials](https://mlsanity.com/paper/2204.11953/) (2022)
- [ChemSpaceAL: An Efficient Active Learning Methodology Applied to Protein-Specific Molecular Generation](https://mlsanity.com/paper/2309.05853/) (2023)
- [Difficulty in chirality recognition for Transformer architectures learning chemical structures from string](https://mlsanity.com/paper/2303.11593/) (2023)
- [LLamol: A Dynamic Multi-Conditional Generative Transformer for De Novo Molecular Design](https://mlsanity.com/paper/2311.14407/) (2023)
- [A novel molecule generative model of VAE combined with Transformer for unseen structure generation](https://mlsanity.com/paper/2402.11950/) (2024)
- [A Large Language Model-based multi-agent manufacturing system for intelligent shopfloor](https://mlsanity.com/paper/2405.16887/) (2024)
- [Investigation of a Data Split Strategy Involving the Time Axis in Adverse Event Prediction Using Machine Learning](https://mlsanity.com/paper/2204.08682/) (2022)
- [Materials Transformers Language Models for Generative Materials Design: a benchmark study](https://mlsanity.com/paper/2206.13578/) (2022)
- [Hybrid Generative AI for De Novo Design of Co-Crystals with Enhanced Tabletability](https://mlsanity.com/paper/2410.17005/) (2024)
- [Investigation of chemical structure recognition by encoder-decoder models in learning progress](https://mlsanity.com/paper/2210.16307/) (2022)
- [Scaffold-Based Multi-Objective Drug Candidate Optimization](https://mlsanity.com/paper/2301.07175/) (2022)
- [CardioGenAI: A Machine Learning-Based Framework for Re-Engineering Drugs for Reduced hERG Liability](https://mlsanity.com/paper/2403.07632/) (2024)
- [The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?](https://mlsanity.com/paper/2501.13952/) (2025)
- [ChemFixer: Correcting Invalid Molecules to Unlock Previously Unseen Chemical Space](https://mlsanity.com/paper/2511.13758/) (2025)

## Abstract (arXiv)

Discovering new materials better suited to specific purposes is an important issue in improving the quality of human life. Here, a neural network that creates molecules that meet some desired conditions based on a deep understanding of chemical language is proposed (Generative Chemical Transformer, GCT). The attention mechanism in GCT allows a deeper understanding of molecular structures beyond the limitations of chemical language itself which cause semantic discontinuity by paying attention to characters sparsely. It is investigated that the significance of language models for inverse molecular design problems by quantitatively evaluating the quality of the generated molecules. GCT generates highly realistic chemical strings that satisfy both chemical and linguistic grammar rules. Molecules parsed from generated strings simultaneously satisfy the multiple target properties and vary for a single condition set. These advances will contribute to improving the quality of human life by accelerating the process of desired material discovery.