# Exploring evolution-aware & -free protein language models as protein function predictors

arXiv: https://arxiv.org/abs/2206.06583 · HTML: https://mlsanity.com/paper/2206.06583/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 59 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [Learning protein sequence embeddings using information from structure](https://mlsanity.com/paper/1902.08661/) (2019)
- [Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model](https://mlsanity.com/paper/1609.00680/) (2016)
- [Aligning 415 519 proteins in less than two hours on PC](https://mlsanity.com/paper/1603.06958/) (2016)
- [Protein Secondary Structure Prediction with Long Short Term Memory Networks](https://mlsanity.com/paper/1412.7828/) (2014)

## Built on by (most cited first, 20 of 20)

- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [Structure-informed Language Models Are Protein Designers](https://mlsanity.com/paper/2302.01649/) (2023)
- [InstructProtein: Aligning Human and Protein Language via Knowledge Instruction](https://mlsanity.com/paper/2310.03269/) (2023)
- [Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trials](https://mlsanity.com/paper/2409.04481/) (2024)
- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [Endowing Protein Language Models with Structural Knowledge](https://mlsanity.com/paper/2401.14819/) (2024)
- [Retrieved Sequence Augmentation for Protein Representation Learning](https://mlsanity.com/paper/2302.12563/) (2023)
- [Simple, Efficient and Scalable Structure-aware Adapter Boosts Protein Language Models](https://mlsanity.com/paper/2404.14850/) (2024)
- [PFMBench: Protein Foundation Model Benchmark](https://mlsanity.com/paper/2506.14796/) (2025)
- [VenusFactory: A Unified Platform for Protein Engineering Data Retrieval and Language Model Fine-Tuning](https://mlsanity.com/paper/2503.15438/) (2025)
- [OneProt: Towards Multi-Modal Protein Foundation Models](https://mlsanity.com/paper/2411.04863/) (2024)
- [Interpretable Enzyme Function Prediction via Residue-Level Detection](https://mlsanity.com/paper/2501.05644/) (2025)
- [Pairing interacting protein sequences using masked language modeling](https://mlsanity.com/paper/2308.07136/) (2023)
- [Exploring Post-Training Quantization of Protein Language Models](https://mlsanity.com/paper/2310.19624/) (2023)
- [Boosting Protein Language Models with Negative Sample Mining](https://mlsanity.com/paper/2405.17902/) (2024)
- [Structure-Aligned Protein Language Model](https://mlsanity.com/paper/2505.16896/) (2025)
- [Discriminative protein sequence modelling with Latent Space Diffusion](https://mlsanity.com/paper/2503.18551/) (2025)
- [Do Protein Transformers Have Biological Intelligence?](https://mlsanity.com/paper/2506.06701/) (2025)
- [Demystify Protein Generation with Hierarchical Conditional Diffusion Models](https://mlsanity.com/paper/2507.18603/) (2025)
- [SurfDesign: Effective Protein Design on Molecular Surfaces](https://mlsanity.com/paper/2606.07567/) (2026)

## Abstract (arXiv)

Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions. In particular, AlphaFold, a ground-breaking AI system, could potentially reshape structural biology. However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction. In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence), MSA-Transformer (multiple sequence alignment) and Evoformer (structural), with a special focus on Evoformer. Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of AlphaFold produce representations amenable to predicting protein function? (ii) If yes, can Evoformer replace ESM-1b and MSA-Transformer? (ii) How much do these PLMs rely on evolution-related protein data? In this regard, are they complementary to each other? We compare these models by empirical study along with new insights and conclusions. All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs.