mlsanityData to 14 Oct 2026

A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design

arXiv 2507.13646 ↗First posted 18 Jul 2025cs.LG cs.AI q-bio.QMcited by 1

Not on the map. Why?

Abstract (arXiv)

The impact of Transformer-based language models has been unprecedented in Natural Language Processing (NLP). The success of such models has also led to their adoption in other fields including bioinformatics. Taking this into account, this paper discusses recent advances in Transformer-based models for protein sequence analysis and design. In this review, we have discussed and analysed a significant number of works pertaining to such applications. These applications encompass gene ontology, functional and structural protein identification, generation of de novo proteins and binding of proteins. We attempt to shed light on the strength and weaknesses of the discussed works to provide a comprehensive insight to readers. Finally, we highlight shortcomings in existing research and explore potential avenues for future developments. We believe that this review will help researchers working in this field to have an overall idea of the state of the art in this field, and to orient their future studies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 109 references that are in the map. First posted on arXiv 18 Jul 2025.

Builds on

24 of 109 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
12 more
  1. 2019Evaluating Protein Transfer Learning with TAPEcited by 231
  2. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 159
  3. 2020ProGen: Language Modeling for Protein Generationcited by 128
  4. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  5. 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
  6. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  7. 2022Regression Transformer: Concurrent sequence regression and generation for molecular language modelingcited by 50
  8. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 50
  9. 2024Uni-Mol2: Exploring Molecular Pretraining Model at Scalecited by 20
  10. 2025CFP-Gen: Combinatorial Functional Protein Generation via Diffusion Language Modelscited by 8
  11. 2024A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysiscited by 1
  12. 2025Bidirectional Hierarchical Protein Multi-Modal Representation Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Heterogeneous Scientific Foundation Model Collaborationcited by 1