# SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision

arXiv: https://arxiv.org/abs/2412.05569 · HTML: https://mlsanity.com/paper/2412.05569/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 7 Dec 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [Encode, Tag, Realize: High-Precision Text Editing](https://mlsanity.com/paper/1909.01187/) (2019)
- [N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules](https://mlsanity.com/paper/1806.09206/) (2018)
- [MolXPT: Wrapping Molecules with Text for Generative Pre-training](https://mlsanity.com/paper/2305.10688/) (2023)
- [Felix: Flexible Text Editing Through Tagging and Insertion](https://mlsanity.com/paper/2003.10687/) (2020)
- ["Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Models](https://mlsanity.com/paper/1711.04810/) (2017)
- [Lexically Constrained Neural Machine Translation with Levenshtein Transformer](https://mlsanity.com/paper/2004.12681/) (2020)
- [A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining](https://mlsanity.com/paper/2305.18407/) (2023)
- [Contextual Representation Learning beyond Masked Language Modeling](https://mlsanity.com/paper/2204.04163/) (2022)
- [ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling](https://mlsanity.com/paper/2403.12995/) (2024)

## Built on by (most cited first, 3 of 3)

- [ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models](https://mlsanity.com/paper/2501.13397/) (2025)
- [Self-Supervised Learning on Molecular Graphs: A Systematic Investigation of Masking Design](https://mlsanity.com/paper/2512.07064/) (2025)
- [ProbeMatchDTI: Probe-Driven Multi-Scale Biochemical Pattern Matching for Drug-Target Interaction Prediction](https://mlsanity.com/paper/2609.02549/) (2026)

## Abstract (arXiv)

SMILES, a crucial textual representation of molecular structures, has garnered significant attention as a foundation for pre-trained language models (LMs). However, most existing pre-trained SMILES LMs focus solely on the single-token level supervision during pre-training, failing to fully leverage the substructural information of molecules. This limitation makes the pre-training task overly simplistic, preventing the models from capturing richer molecular semantic information. Moreover, during pre-training, these SMILES LMs only process corrupted SMILES inputs, never encountering any valid SMILES, which leads to a train-inference mismatch. To address these challenges, we propose SMI-Editor, a novel edit-based pre-trained SMILES LM. SMI-Editor disrupts substructures within a molecule at random and feeds the resulting SMILES back into the model, which then attempts to restore the original SMILES through an editing process. This approach not only introduces fragment-level training signals, but also enables the use of valid SMILES as inputs, allowing the model to learn how to reconstruct complete molecules from these incomplete structures. As a result, the model demonstrates improved scalability and an enhanced ability to capture fragment-level molecular information. Experimental results show that SMI-Editor achieves state-of-the-art performance across multiple downstream molecular tasks, and even outperforming several 3D molecular representation models.