mlsanityData to 7 Oct 2026

Could Chemical LLMs benefit from Message Passing

arXiv 2405.08334 ↗First posted 14 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretrained language models (LMs) showcase significant capabilities in processing molecular text, while concurrently, message passing neural networks (MPNNs) demonstrate resilience and versatility in the domain of molecular science. Despite these advancements, we find there are limited studies investigating the bidirectional interactions between molecular structures and their corresponding textual representations. Therefore, in this paper, we propose two strategies to evaluate whether an information integration can enhance the performance: contrast learning, which involves utilizing an MPNN to supervise the training of the LM, and fusion, which exploits information from both models. Our empirical analysis reveals that the integration approaches exhibit superior performance compared to baselines when applied to smaller molecular graphs, while these integration approaches do not yield performance enhancements on large scale graphs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

23 of 34 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  7. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  8. 2020Directional Message Passing for Molecular Graphscited by 556
  9. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  10. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  11. 2021Graph Contrastive Learning Automatedcited by 287
  12. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
11 more
  1. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  2. 2019Evaluating Protein Transfer Learning with TAPEcited by 229
  3. 2020Transformer Networks for Trajectory Forecastingcited by 180
  4. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  5. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  6. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 89
  7. 2022PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understandingcited by 77
  8. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  9. 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 47
  10. 2021Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Modelcited by 20
  11. 2021Speech Recognition by Simply Fine-tuning BERTcited by 11

Built on by

Not cited yet.