# Training Text-to-Molecule Models with Context-Aware Tokenization

arXiv: https://arxiv.org/abs/2509.04476 · HTML: https://mlsanity.com/paper/2509.04476/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 30 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Hierarchical Generation of Molecular Graphs using Structural Motifs](https://mlsanity.com/paper/2002.03230/) (2020)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [Unifying Molecular and Textual Representations via Multi-task Language Modelling](https://mlsanity.com/paper/2301.12586/) (2023)
- [Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective](https://mlsanity.com/paper/2306.06615/) (2023)
- [MolXPT: Wrapping Molecules with Text for Generative Pre-training](https://mlsanity.com/paper/2305.10688/) (2023)
- [Molecule Generation by Principal Subgraph Mining and Assembling](https://mlsanity.com/paper/2106.15098/) (2021)
- [Sample-efficient Multi-objective Molecular Optimization with GFlowNets](https://mlsanity.com/paper/2302.04040/) (2023)
- [Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules](https://mlsanity.com/paper/2310.14753/) (2023)
- [De Novo Molecular Generation via Connection-aware Motif Mining](https://mlsanity.com/paper/2302.01129/) (2023)
- [A Deep Generative Model for Molecule Optimization via One Fragment Modification](https://mlsanity.com/paper/2012.04231/) (2020)
- [Text-Guided Molecule Generation with Diffusion Language Model](https://mlsanity.com/paper/2402.13040/) (2024)
- [Fragment-based Pretraining and Finetuning on Molecular Graphs](https://mlsanity.com/paper/2310.03274/) (2023)
- [t-SMILES: A Scalable Fragment-based Molecular Representation Framework for De Novo Molecule Generation](https://mlsanity.com/paper/2301.01829/) (2023)
- [From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery](https://mlsanity.com/paper/2309.05203/) (2023)
- [LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models](https://mlsanity.com/paper/2405.17829/) (2024)
- [3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation](https://mlsanity.com/paper/2403.07179/) (2024)
- [Data-Efficient Molecular Generation with Hierarchical Textual Inversion](https://mlsanity.com/paper/2405.02845/) (2024)

## Built on by (most cited first, 1 of 1)

- [BiMol-Diff: A Unified Diffusion Framework for Molecular Generation and Captioning](https://mlsanity.com/paper/2604.24089/) (2026)

## Abstract (arXiv)

Recently, text-to-molecule models have shown great potential across various chemical applications, e.g., drug-discovery. These models adapt language models to molecular data by representing molecules as sequences of atoms. However, they rely on atom-level tokenizations, which primarily focus on modeling local connectivity, thereby limiting the ability of models to capture the global structural context within molecules. To tackle this issue, we propose a novel text-to-molecule model, coined Context-Aware Molecular T5 (CAMT5). Inspired by the significance of the substructure-level contexts in understanding molecule structures, e.g., ring systems, we introduce substructure-level tokenization for text-to-molecule models. Building on our tokenization scheme, we develop an importance-based training strategy that prioritizes key substructures, enabling CAMT5 to better capture the molecular semantics. Extensive experiments verify the superiority of CAMT5 in various text-to-molecule generation tasks. Intriguingly, we find that CAMT5 outperforms the state-of-the-art methods using only 2% of training tokens. In addition, we propose a simple yet effective ensemble strategy that aggregates the outputs of text-to-molecule models to further boost the generation performance. Code is available at https://github.com/Songhyeontae/CAMT5.git.