mlsanityData to 14 Oct 2026

Thinking like a CHEMIST: Combined Heterogeneous Embedding Model Integrating Structure and Tokens

arXiv 2502.17986 ↗First posted 25 Feb 2025cs.LG cs.AIcited by 0

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Representing molecular structures effectively in chemistry remains a challenging task. Language models and graph-based models are extensively utilized within this domain, consistently achieving state-of-the-art results across an array of tasks. However, the prevailing practice of representing chemical compounds in the SMILES format - used by most data sets and many language models - presents notable limitations as a training data format. In this study, we present a novel approach that decomposes molecules into substructures and computes descriptor-based representations for these fragments, providing more detailed and chemically relevant input for model training. We use this substructure and descriptor data as input for language model and also propose a bimodal architecture that integrates this language model with graph-based models. As LM we use RoBERTa, Graph Isomorphism Networks (GIN), Graph Convolutional Networks (GCN) and Graphormer as graph ones. Our framework shows notable improvements over traditional methods in various tasks such as Quantitative Structure-Activity Relationship (QSAR) prediction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 14 of its 69 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

14 of 69 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2017Graph Attention Networkscited by 8,923
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  7. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  8. 2022Transformers in Medical Imaging: A Surveycited by 245
  9. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  10. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  11. 2022Graph neural networks for materials science and chemistrycited by 182
  12. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 127
2 more
  1. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  2. 2021Using Molecular Embeddings in QSAR Modeling: Does it Make a Difference?cited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GLACIER: A Multimodal Student-Teacher Foundation Model for Molecular Property Prediction
  2. 2024LLM and GNN are Complementary: Distilling LLM for Multimodal Graph Learning
  3. 2024Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs
  4. 2025Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoders
  5. 2025M2LLM: Multi-view Molecular Representation Learning with Large Language Models
  6. 2026VQ-Atom: Semantic Discretization of Local Atomic Environments for Molecular Representation Learning
  7. 2026MoleCode unlocks structural intelligence in large language models
  8. 2026What Does a Chemical Language Model Know About Molecules?