mlsanityData to 7 Oct 2026

ProtChatGPT: Towards Understanding Proteins with Large Language Models

arXiv 2402.09649 ↗First posted 15 Feb 2024cs.CE cs.AI q-bio.BMcited by 18

Not on the map. Why?

Abstract (arXiv)

Protein research is crucial in various fundamental disciplines, but understanding their intricate structure-function relationships remains challenging. Recent Large Language Models (LLMs) have made significant strides in comprehending task-specific knowledge, suggesting the potential for ChatGPT-like systems specialized in protein to facilitate basic research. In this work, we introduce ProtChatGPT, which aims at learning and understanding protein structures via natural languages. ProtChatGPT enables users to upload proteins, ask questions, and engage in interactive conversations to produce comprehensive answers. The system comprises protein encoders, a Protein-Language Pertaining Transformer (PLP-former), a projection adapter, and an LLM. The protein first undergoes protein encoders and PLP-former to produce protein embeddings, which are then projected by the adapter to conform with the LLM. The LLM finally combines user questions with projected embeddings to generate informative answers. Experiments show that ProtChatGPT can produce promising responses to proteins and their corresponding questions. We hope that ProtChatGPT could form the basis for further exploration and application in protein research. Code and our pre-trained model will be publicly available.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

36 of 57 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
24 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2022Galactica: A Large Language Model for Sciencecited by 736
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 308
  13. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  14. 2022Protein Representation Learning by Geometric Structure Pretrainingcited by 161
  15. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  16. 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
  17. 2021Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-designcited by 82
  18. 2022Tranception: protein fitness prediction with autoregressive transformers and inference-time retrievalcited by 78
  19. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  20. 2022Conditional Antibody Design as 3D Equivariant Graph Translationcited by 75
  21. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  22. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  23. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
  24. 2022VL-BEiT: Generative Vision-Language Pretrainingcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 4.

18 papers.

Sort:

By area:

  1. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  2. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  3. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 16
  4. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  5. 2024TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineeringcited by 9
  6. 2024EvoLlama: Enhancing LLMs' Understanding of Proteins via Multimodal Structure and Sequence Representationscited by 9
  7. 2025Computational Protein Science in the Era of Large Language Models (LLMs)cited by 6
  8. 2024Unifying Sequences, Structures, and Descriptions for Any-to-Any Protein Generation with the Large Multimodal Model HelixProtXcited by 3
  9. 2024Aligning Large Language Models and Geometric Deep Models for Protein Representationcited by 3
  10. 2025ProtTeX: Structure-In-Context Reasoning and Editing of Proteins with Large Language Modelscited by 3
  11. 2025Prot2Chat: Protein LLM with Early-Fusion of Text, Sequence and Structurecited by 2
  12. 2025Speak to a Protein: An Interactive Multimodal Co-Scientistcited by 1
6 more
  1. 2025Protein as a Second Language for LLMscited by 0
  2. 2025Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advancescited by 0
  3. 2026Towards Multiscale Graph-based Protein Learning with Geometric Secondary Structural Motifscited by 0
  4. 2026Scaling-Aware Adapter for Structure-Grounded LLM Reasoningcited by 0
  5. 2026ES-Merging: Biological MLLM Merging via Embedding Space Signalscited by 0
  6. 2026Protein Thoughts: Interpretable Reasoning with Tree of Thoughts and Embedding-Space Flow Matching for Protein-Protein Interaction Discoverycited by 0