# General Protein Pretraining or Domain-Specific Designs? Benchmarking Protein Modeling on Realistic Applications

arXiv: https://arxiv.org/abs/2506.02052 · HTML: https://mlsanity.com/paper/2506.02052/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 75 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval](https://mlsanity.com/paper/2205.13760/) (2022)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [One Transformer Can Understand Both 2D & 3D Molecular Data](https://mlsanity.com/paper/2210.01765/) (2022)
- [OntoProtein: Protein Pretraining With Gene Ontology Embedding](https://mlsanity.com/paper/2201.11147/) (2022)
- [ProteinBench: A Holistic Evaluation of Protein Foundation Models](https://mlsanity.com/paper/2409.06744/) (2024)
- [Evaluating representation learning on the protein structure universe](https://mlsanity.com/paper/2406.13864/) (2024)
- [Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information](https://mlsanity.com/paper/1912.05625/) (2019)

## Abstract (arXiv)

Recently, extensive deep learning architectures and pretraining strategies have been explored to support downstream protein applications. Additionally, domain-specific models incorporating biological knowledge have been developed to enhance performance in specialized tasks. In this work, we introduce Protap, a comprehensive benchmark that systematically compares backbone architectures, pretraining strategies, and domain-specific models across diverse and realistic downstream protein applications. Specifically, Protap covers five applications: three general tasks and two novel specialized tasks, i.e., enzyme-catalyzed protein cleavage site prediction and targeted protein degradation, which are industrially relevant yet missing from existing benchmarks. For each application, Protap compares various domain-specific models and general architectures under multiple pretraining settings. Our empirical studies imply that: (i) Though large-scale pretraining encoders achieve great results, they often underperform supervised encoders trained on small downstream training sets. (ii) Incorporating structural information during downstream fine-tuning can match or even outperform protein language models pretrained on large-scale sequence corpora. (iii) Domain-specific biological priors can enhance performance on specialized downstream tasks. Code and datasets are publicly available at https://github.com/Trust-App-AI-Lab/protap.