mlsanityData to 14 Oct 2026

Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing

arXiv 2212.10789 ↗First posted 21 Dec 2022cs.LG cs.CL q-bio.QMcited by 166

Not on the map. Why?

Abstract (arXiv)

There is increasing adoption of artificial intelligence in drug discovery. However, existing studies use machine learning to mainly utilize the chemical structures of molecules but ignore the vast textual knowledge available in chemistry. Incorporating textual knowledge enables us to realize new drug design objectives, adapt to text-based instructions and predict complex biological activities. Here we present a multi-modal molecule structure-text model, MoleculeSTM, by jointly learning molecules' chemical structures and textual descriptions via a contrastive learning strategy. To train MoleculeSTM, we construct a large multi-modal dataset, namely, PubChemSTM, with over 280,000 chemical structure-text pairs. To demonstrate the effectiveness and utility of MoleculeSTM, we design two challenging zero-shot tasks based on text instructions, including structure-text retrieval and molecule editing. MoleculeSTM has two main properties: open vocabulary and compositionality via natural language. In experiments, MoleculeSTM obtains the state-of-the-art generalization ability to novel biochemical concepts across various benchmarks.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Builds on 29 of its 87 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

29 of 87 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2018How Powerful are Graph Neural Networks?cited by 4,393
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  11. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
17 more
  1. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  2. 2021E(n) Equivariant Graph Neural Networkscited by 965
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  4. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
  5. 2020Contrastive Multi-View Representation Learning on Graphscited by 652
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  7. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  8. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  9. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  10. 2020Hierarchical Generation of Molecular Graphs using Structural Motifscited by 208
  11. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  12. 2021Geometric Deep Learning on Molecular Representationscited by 113
  13. 2018N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Moleculescited by 93
  14. 2022Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matchingcited by 68
  15. 2022DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotationscited by 54
  16. 2022Retrieval-based Controllable Molecule Generationcited by 44
  17. 2022Evaluating Self-Supervised Learning for Molecular Graph Embeddingscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 71; 2025: 47; 2026: 28.

166 papers.

Sort:

By area:

  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  2. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  3. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 133
  4. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  5. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  6. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  7. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  8. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  9. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  10. 2023Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspectivecited by 92
  11. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  12. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 85
38 more
  1. 2024ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphscited by 85
  2. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  3. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  4. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  5. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  6. 2024A Survey of Large Language Models for Graphscited by 72
  7. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  8. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  9. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 66
  10. 2023A Text-guided Protein Design Frameworkcited by 64
  11. 2024Efficient Evolutionary Search Over Chemical Space with Large Language Modelscited by 61
  12. 2023MolFM: A Multimodal Molecular Foundation Modelcited by 50
  13. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  14. 2024GLBench: A Comprehensive Benchmark for Graph with Large Language Modelscited by 36
  15. 2024ChemReasoner: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedbackcited by 33
  16. 2023A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretrainingcited by 32
  17. 2023ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedbackcited by 31
  18. 2024Text-Guided Molecule Generation with Diffusion Language Modelcited by 31
  19. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  20. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  21. 2024MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Predictioncited by 24
  22. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  23. 2024UniMoT: Unified Molecule-Text Language Model with Discrete Token Representationcited by 23
  24. 2023Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materialscited by 22
  25. 2024MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehensioncited by 22
  26. 2024Mobility-LLM: Learning Visiting Intentions and Travel Preferences from Human Mobility Data with Large Language Modelscited by 22
  27. 2024LLaMo: Large Language Model-based Molecular Graph Assistantcited by 22
  28. 2024L+M-24: Building a Dataset for Language + Molecules @ ACL 2024cited by 21
  29. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  30. 2024ReactXT: Understanding Molecular "Reaction-ship" via Reaction-Contextualized Molecule-Text Pretrainingcited by 19
  31. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  32. 2024A quantitative analysis of knowledge-learning preferences in large language models in molecular sciencecited by 18
  33. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
  34. 2025RAG-Enhanced Collaborative LLM Agents for Drug Discoverycited by 18
  35. 2023Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Designcited by 17
  36. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 17
  37. 2024HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignmentcited by 17
  38. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 17

50 most cited of 166 All 166 (JSON)