mlsanityData to 7 Oct 2026

Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule

arXiv 2403.13830 ↗First posted 7 Mar 2024q-bio.BM cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence has demonstrated immense potential in scientific research. Within molecular science, it is revolutionizing the traditional computer-aided paradigm, ushering in a new era of deep learning. With recent progress in multimodal learning and natural language processing, an emerging trend has targeted at building multimodal frameworks to jointly model molecules with textual domain knowledge. In this paper, we present the first systematic survey on multimodal frameworks for molecules research. Specifically,we begin with the development of molecular deep learning and point out the necessity to involve textual modality. Next, we focus on recent advances in text-molecule alignment methods, categorizing current models into two groups based on their architectures and listing relevant pre-training tasks. Furthermore, we delves into the utilization of large language models and prompting techniques for molecular tasks and present significant applications in drug discovery. Finally, we discuss the limitations in this field and highlight several promising directions for future research.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

51 of 67 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2020Graph Contrastive Learning with Augmentationscited by 1,203
38 more
  1. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  2. 2022Galactica: A Large Language Model for Sciencecited by 736
  3. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  6. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  7. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  8. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  9. 2022Translation between Molecules and Natural Languagecited by 173
  10. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  11. 2023Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Modelscited by 131
  12. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 129
  13. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  14. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  15. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  16. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 92
  17. 2023Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspectivecited by 91
  18. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  19. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  20. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  21. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  22. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  23. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 65
  24. 2023A Text-guided Protein Design Frameworkcited by 61
  25. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  26. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  27. 2023DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule Graphscited by 40
  28. 2023Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformerscited by 32
  29. 2023ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedbackcited by 30
  30. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  31. 2023InstructProtein: Aligning Human and Protein Language via Knowledge Instructioncited by 27
  32. 2023Structured Chemistry Reasoning with Large Language Modelscited by 27
  33. 2023BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphscited by 24
  34. 2023ReLM: Leveraging Language Models for Enhanced Chemical Reaction Predictioncited by 24
  35. 2023From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discoverycited by 11
  36. 2023Predictive Chemistry Augmented with Text Retrievalcited by 9
  37. 2023Towards Unified AI Drug Discovery with Multiple Knowledge Modalitiescited by 4
  38. 2023Extracting human interpretable structure-property relationships in chemistry using XAI and large language modelscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  2. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  3. 2024Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Predictioncited by 4
  4. 2025MolEdit: Knowledge Editing for Multimodal Molecule Language Modelscited by 3