mlsanityData to 14 Oct 2026

Transformers in Protein: A Survey

arXiv 2505.20098 ↗First posted 26 May 2025cs.LG cs.CR q-bio.QMcited by 0

Not on the map. Why?

Abstract (arXiv)

As protein informatics advances rapidly, the demand for enhanced predictive accuracy, structural analysis, and functional understanding has intensified. Transformer models, as powerful deep learning architectures, have demonstrated unprecedented potential in addressing diverse challenges across protein research. However, a comprehensive review of Transformer applications in this field remains lacking. This paper bridges this gap by surveying over 100 studies, offering an in-depth analysis of practical implementations and research progress of Transformers in protein-related tasks. Our review systematically covers critical domains, including protein structure prediction, function prediction, protein-protein interaction analysis, functional annotation, and drug discovery/target identification. To contextualize these advancements across various protein domains, we adopt a domain-oriented classification system. We first introduce foundational concepts: the Transformer architecture and attention mechanisms, categorize Transformer variants tailored for protein science, and summarize essential protein knowledge. For each research domain, we outline its objectives and background, critically evaluate prior methods and their limitations, and highlight transformative contributions enabled by Transformer models. We also curate and summarize pivotal datasets and open-source code resources to facilitate reproducibility and benchmarking. Finally, we discuss persistent challenges in applying Transformers to protein informatics and propose future research directions. This review aims to provide a consolidated foundation for the synergistic integration of Transformer and protein informatics, fostering further innovation and expanded applications in the field.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 99 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

29 of 99 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2017Deformable Convolutional Networkscited by 1,488
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
17 more
  1. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  2. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  3. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  4. 2019Evaluating Protein Transfer Learning with TAPEcited by 231
  5. 2017Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Modelscited by 206
  6. 2020ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 162
  7. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  8. 2020Molecular graph generation with Graph Neural Networkscited by 80
  9. 2018DeepAffinity: Interpretable Deep Learning of Compound-Protein Affinity through Unified Recurrent and Convolutional Neural Networkscited by 52
  10. 2016Deep Learning in Bioinformaticscited by 48
  11. 2023Transformers and Large Language Models for Chemistry and Drug Discoverycited by 13
  12. 2020Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Modelscited by 12
  13. 2024Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detectioncited by 11
  14. 2023PMANet: Malicious URL detection via post-trained language model guided multi-level feature attention networkcited by 10
  15. 2024Ethereum Fraud Detection via Joint Transaction Language Model and Graph Representation Learningcited by 8
  16. 2024A Protein Structure Prediction Approach Leveraging Transformer and CNN Integrationcited by 4
  17. 2023Insights Into the Inner Workings of Transformer Models for Protein Function Predictioncited by 2

Built on by

Not cited yet.