# Chemical Language Models for Natural Products: A State-Space Model Approach

arXiv: https://arxiv.org/abs/2602.13958 · HTML: https://mlsanity.com/paper/2602.13958/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 75 references that are in the map.
- First posted on arXiv 15 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [The Synthesizability of Molecules Proposed by Generative Models](https://mlsanity.com/paper/2002.07007/) (2020)
- [Getting the most out of your tokenizer for pre-training and domain adaptation](https://mlsanity.com/paper/2402.01035/) (2024)
- [LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement](https://mlsanity.com/paper/2504.16053/) (2025)
- [UU-Mamba: Uncertainty-aware U-Mamba for Cardiac Image Segmentation](https://mlsanity.com/paper/2405.17496/) (2024)
- [From Words to Molecules: A Survey of Large Language Models in Chemistry](https://mlsanity.com/paper/2402.01439/) (2024)
- [SMILES-Mamba: Chemical Mamba Foundation Models for Drug ADMET Prediction](https://mlsanity.com/paper/2408.05696/) (2024)
- [Transformers for molecular property prediction: Lessons learned from the past five years](https://mlsanity.com/paper/2404.03969/) (2024)
- [Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods](https://mlsanity.com/paper/2411.17669/) (2024)
- [Transformers for molecular property prediction: Domain adaptation efficiently improves performance](https://mlsanity.com/paper/2503.03360/) (2025)
- [Mamba State-Space Models Are Lyapunov-Stable Learners](https://mlsanity.com/paper/2406.00209/) (2024)
- [Transfer Learning for Molecular Property Predictions from Small Data Sets](https://mlsanity.com/paper/2404.13393/) (2024)
- [NPGPT: Natural Product-Like Compound Generation with GPT-based Chemical Language Models](https://mlsanity.com/paper/2411.12886/) (2024)

## Abstract (arXiv)

Language models are widely used in chemistry for molecular property prediction and small-molecule generation, yet Natural Products (NPs) remain underexplored despite their importance in drug discovery. To address this gap, we develop NP-specific chemical language models (NPCLMs) by pre-training state-space models (Mamba and Mamba-2) and comparing them with transformer baselines (GPT). Using a dataset of about 1M NPs, we present the first systematic comparison of selective state-space models and transformers for NP-focused tasks, together with eight tokenization strategies including character-level, Atom-in-SMILES (AIS), byte-pair encoding (BPE), and NP-specific BPE. We evaluate molecule generation (validity, uniqueness, novelty) and property prediction (membrane permeability, taste, anti-cancer activity) using MCC and AUC-ROC. Mamba generates 1-2 percent more valid and unique molecules than Mamba-2 and GPT, with fewer long-range dependency errors, while GPT yields slightly more novel structures. For property prediction, Mamba variants outperform GPT by 0.02-0.04 MCC under random splits, while scaffold splits show comparable performance. Results demonstrate that domain-specific pre-training on about 1M NPs can match models trained on datasets over 100 times larger.