mlsanityData to 7 Oct 2026

Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey

arXiv 2403.01528 ↗First posted 3 Mar 2024cs.CL cs.AI q-bio.BMcited by 16

Not on the map. Why?

Abstract (arXiv)

The integration of biomolecular modeling with natural language (BL) has emerged as a promising interdisciplinary area at the intersection of artificial intelligence, chemistry and biology. This approach leverages the rich, multifaceted descriptions of biomolecules contained within textual data sources to enhance our fundamental understanding and enable downstream computational tasks such as biomolecule property prediction. The fusion of the nuanced narratives expressed through natural language with the structural and functional specifics of biomolecules described via various molecular modeling techniques opens new avenues for comprehensively representing and analyzing biomolecules. By incorporating the contextual language data that surrounds biomolecules into their modeling, BL aims to capture a holistic view encompassing both the symbolic qualities conveyed through language as well as quantitative structural characteristics. In this review, we provide an extensive analysis of recent advancements achieved through cross modeling of biomolecules and natural language. (1) We begin by outlining the technical representations of biomolecules employed, including sequences, 2D graphs, and 3D structures. (2) We then examine in depth the rationale and key objectives underlying effective multi-modal integration of language and molecular data sources. (3) We subsequently survey the practical applications enabled to date in this developing research area. (4) We also compile and summarize the available resources and datasets to facilitate future work. (5) Looking ahead, we identify several promising research directions worthy of further exploration and investment to continue advancing the field. The related resources and contents are updating in https://github.com/QizhiPei/Awesome-Biomolecule-Language-Cross-Modeling.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 126 of its 387 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

126 of 387 references.

References · 126

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  12. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  13. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  14. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  15. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  16. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  17. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  18. 2019Publicly Available Clinical BERT Embeddingscited by 995
  19. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  20. 2022Galactica: A Large Language Model for Sciencecited by 736
  21. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  22. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  23. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  24. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  25. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  26. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  27. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  28. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  29. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 308
  30. 2023Tool Learning with Foundation Modelscited by 302
  31. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  32. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  33. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  34. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  35. 2023Generating Images with Multimodal Language Modelscited by 259
  36. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  37. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  38. 2019Evaluating Protein Transfer Learning with TAPEcited by 229

50 most cited of 126 All 126 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  3. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  4. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  5. 2025RAG-Enhanced Collaborative LLM Agents for Drug Discoverycited by 18
  6. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  7. 20243D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modelingcited by 9
  8. 2024SciDFM: A Large Language Model with Mixture-of-Experts for Sciencecited by 7
  9. 2024Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systemscited by 6
  10. 2025Computational Protein Science in the Era of Large Language Models (LLMs)cited by 6
  11. 2025KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledgecited by 6
  12. 2026LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoningcited by 6
4 more
  1. 2024Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding with LLMscited by 4
  2. 2025MolEdit: Knowledge Editing for Multimodal Molecule Language Modelscited by 3
  3. 2026AMix-2: Establishing Protein as a Native Modality in Large Language Modelscited by 1
  4. 2024Chemical Language Model Linker: blending text and molecules with modular adapterscited by 0