# Pre-trained Molecular Language Models with Random Functional Group Masking

arXiv: https://arxiv.org/abs/2411.01401 · HTML: https://mlsanity.com/paper/2411.01401/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)
- [Geometric Deep Learning on Molecular Representations](https://mlsanity.com/paper/2107.12375/) (2021)
- [Dual-view Molecule Pre-training](https://mlsanity.com/paper/2106.10234/) (2021)
- [Learn molecular representations from large-scale unlabeled molecules for drug discovery](https://mlsanity.com/paper/2012.11175/) (2020)
- [Self-Supervised Graph Transformer on Large-Scale Molecular Data](https://mlsanity.com/paper/2007.02835/) (2020)
- [Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction](https://mlsanity.com/paper/2207.02724/) (2022)
- [G-MATT: Single-step Retrosynthesis Prediction using Molecular Grammar Tree Transformer](https://mlsanity.com/paper/2305.03153/) (2023)

## Built on by (most cited first, 2 of 2)

- [Structure-Aware Fusion with Progressive Injection for Multimodal Molecular Representation Learning](https://mlsanity.com/paper/2510.23640/) (2025)
- [Spatial-Temporal Transformer with Curriculum Learning for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2507.14698/) (2025)

## Abstract (arXiv)

Recent advancements in computational chemistry have leveraged the power of trans-former-based language models, such as MoLFormer, pre-trained using a vast amount of simplified molecular-input line-entry system (SMILES) sequences, to understand and predict molecular properties and activities, a critical step in fields like drug discovery and materials science. To further improve performance, researchers have introduced graph neural networks with graph-based molecular representations, such as GEM, incorporating the topology, geometry, 2D or even 3D structures of molecules into pre-training. While most of molecular graphs in existing studies were automatically converted from SMILES sequences, it is to assume that transformer-based language models might be able to implicitly learn structure-aware representations from SMILES sequences. In this paper, we propose ours -- a SMILES-based underlineem Molecular underlineem Language underlineem Model, which randomly masking SMILES subsequences corresponding to specific molecular underlineem Functional underlineem Groups to incorporate structure information of atoms during the pre-training phase. This technique aims to compel the model to better infer molecular structures and properties, thus enhancing its predictive capabilities. Extensive experimental evaluations across 11 benchmark classification and regression tasks in the chemical domain demonstrate the robustness and superiority of ours. Our findings reveal that ours outperforms existing pre-training models, either based on SMILES or graphs, in 9 out of the 11 downstream tasks, ranking as a close second in the remaining ones.