# Difficulty in chirality recognition for Transformer architectures learning chemical structures from string

arXiv: https://arxiv.org/abs/2303.11593 · HTML: https://mlsanity.com/paper/2303.11593/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 49 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Molecule Attention Transformer](https://mlsanity.com/paper/2002.08264/) (2020)
- [SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery](https://mlsanity.com/paper/1911.04738/) (2019)
- [Path-Augmented Graph Transformer Network](https://mlsanity.com/paper/1905.12712/) (2019)
- [Self-Attention Based Molecule Representation for Predicting Drug-Target Interaction](https://mlsanity.com/paper/1908.06760/) (2019)
- [Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention](https://mlsanity.com/paper/2103.00213/) (2021)
- [Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders](https://mlsanity.com/paper/1806.09300/) (2018)
- [Controlled Molecule Generator for Optimizing Multiple Chemical Properties](https://mlsanity.com/paper/2010.13908/) (2020)
- [Investigation of a Data Split Strategy Involving the Time Axis in Adverse Event Prediction Using Machine Learning](https://mlsanity.com/paper/2204.08682/) (2022)

## Built on by (most cited first, 7 of 7)

- [ProtoMol: Enhancing Molecular Property Prediction via Prototype-Guided Multimodal Learning](https://mlsanity.com/paper/2510.16824/) (2025)
- [A novel molecule generative model of VAE combined with Transformer for unseen structure generation](https://mlsanity.com/paper/2402.11950/) (2024)
- [Learning Molecular Chirality via Chiral Determinant Kernels](https://mlsanity.com/paper/2602.07415/) (2026)
- [Hierarchical Structure Enhances the Convergence and Generalizability of Linear Molecular Representation](https://mlsanity.com/paper/2402.02164/) (2024)
- [MOLAR: Learning Multimodal Molecular Representations from Noisy Labels](https://mlsanity.com/paper/2606.18390/) (2026)
- [From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models](https://mlsanity.com/paper/2605.09949/) (2026)
- [GSF-chi: Global Stereochemical Fields for Chiral Graph Transformers](https://mlsanity.com/paper/2609.12532/) (2026)

## Abstract (arXiv)

Recent years have seen rapid development of descriptor generation based on representation learning of extremely diverse molecules, especially those that apply natural language processing (NLP) models to SMILES, a literal representation of molecular structure. However, little research has been done on how these models understand chemical structure. To address this black box, we investigated the relationship between the learning progress of SMILES and chemical structure using a representative NLP model, the Transformer. We show that while the Transformer learns partial structures of molecules quickly, it requires extended training to understand overall structures. Consistently, the accuracy of molecular property predictions using descriptors generated from models at different learning steps was similar from the beginning to the end of training. Furthermore, we found that the Transformer requires particularly long training to learn chirality and sometimes stagnates with low performance due to misunderstanding of enantiomers. These findings are expected to deepen the understanding of NLP models in chemistry.