mlsanityData to 14 Oct 2026

Tokenization for Molecular Foundation Models

arXiv 2409.15370 ↗First posted 19 Sep 2024cs.LG cs.AI physics.chem-phcited by 5

Not on the map. Why?

Abstract (arXiv)

Text-based foundation models have become an important part of scientific discovery, with molecular foundation models accelerating advancements in material science and molecular design.However, existing models are constrained by closed-vocabulary tokenizers that capture only a fraction of molecular space. In this work, we systematically evaluate 34 tokenizers, including 19 chemistry-specific ones, and reveal significant gaps in their coverage of the SMILES molecular representation. To assess the impact of tokenizer choice, we introduce n-gram language models as a low-cost proxy and validate their effectiveness by pretraining and finetuning 18 RoBERTa-style encoders for molecular property prediction. To overcome the limitations of existing tokenizers, we propose two new tokenizers -- Smirk and Smirk-GPE -- with full coverage of the OpenSMILES specification. The proposed tokenizers systematically integrate nuclear, electronic, and geometric degrees of freedom; facilitating applications in pharmacology, agriculture, biology, and energy storage. Our results highlight the need for open-vocabulary modeling and chemically diverse benchmarks in cheminformatics.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 71 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

36 of 71 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  8. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  11. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  12. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 229
24 more
  1. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  3. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  4. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 127
  5. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 115
  6. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  7. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  8. 2020Mapping the Space of Chemical Reactions Using Attention-Based Neural Networkscited by 70
  9. 2021Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction predictioncited by 68
  10. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  11. 2017"Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Modelscited by 55
  12. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 46
  13. 2022TransPolymer: a Transformer-based language model for polymer property predictionscited by 43
  14. 2023SELFormer: Molecular Representation Learning via SELFIES Language Modelscited by 40
  15. 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
  16. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  17. 2020Stolen Probability: A Structural Weakness of Neural Language Modelscited by 22
  18. 2023ReactionT5: a large-scale pre-trained model towards application of limited reaction datacited by 13
  19. 2023Recent advances in the Self-Referencing Embedding Strings (SELFIES) librarycited by 8
  20. 2024Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challengecited by 8
  21. 2020AutoMat: Accelerated Computational Electrochemical systems Discoverycited by 5
  22. 2023Machine Learning Small Molecule Properties in Drug Discoverycited by 5
  23. 2024A Large Encoder-Decoder Family of Foundation Models For Chemical Languagecited by 5
  24. 2023Accurate Surface and Finite Temperature Bulk Properties of Lithium Metal at Large Scales using Machine Learning Interaction Potentialscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Defining Foundation Models for Computational Science: A Call for Clarity and Rigorcited by 17
  2. 2025NovoMolGen: Rethinking Molecular Language Model Pretrainingcited by 0
  3. 2026BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learningcited by 0
  4. 2026Where to cut, how deep: BPE and Unigram-LM on chemistry SMILEScited by 0
  5. 2026Does Pretraining Teach Chemical Language Models Chemistry?cited by 0