# Simple, Efficient and Scalable Structure-aware Adapter Boosts Protein Language Models

arXiv: https://arxiv.org/abs/2404.14850 · HTML: https://mlsanity.com/paper/2404.14850/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 62 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 15 of 15)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [TorchDrug: A Powerful and Flexible Machine Learning Platform for Drug Discovery](https://mlsanity.com/paper/2202.08320/) (2022)
- [Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling](https://mlsanity.com/paper/2301.06568/) (2023)
- [Is Transfer Learning Necessary for Protein Landscape Prediction?](https://mlsanity.com/paper/2011.03443/) (2020)
- [Exploring evolution-aware & -free protein language models as protein function predictors](https://mlsanity.com/paper/2206.06583/) (2022)
- [Endowing Protein Language Models with Structural Knowledge](https://mlsanity.com/paper/2401.14819/) (2024)
- [Structure-Informed Protein Language Model](https://mlsanity.com/paper/2402.05856/) (2024)
- [SESNet: sequence-structure feature-integrated deep learning method for data-efficient protein engineering](https://mlsanity.com/paper/2301.00004/) (2022)

## Built on by (most cited first, 6 of 6)

- [VenusX: Unlocking Fine-Grained Functional Understanding of Proteins](https://mlsanity.com/paper/2505.11812/) (2025)
- [Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection](https://mlsanity.com/paper/2410.02647/) (2024)
- [Sequence-Only Prediction of Binding Affinity Changes: A Robust and Interpretable Model for Antibody Engineering](https://mlsanity.com/paper/2505.20301/) (2025)
- [OneProt: Towards Multi-Modal Protein Foundation Models](https://mlsanity.com/paper/2411.04863/) (2024)
- [CPE-Pro: A Structure-Sensitive Deep Learning Method for Protein Representation and Origin Evaluation](https://mlsanity.com/paper/2410.15592/) (2024)
- [SurfDesign: Effective Protein Design on Molecular Surfaces](https://mlsanity.com/paper/2606.07567/) (2026)

## Abstract (arXiv)

Fine-tuning Pre-trained protein language models (PLMs) has emerged as a prominent strategy for enhancing downstream prediction tasks, often outperforming traditional supervised learning approaches. As a widely applied powerful technique in natural language processing, employing Parameter-Efficient Fine-Tuning techniques could potentially enhance the performance of PLMs. However, the direct transfer to life science tasks is non-trivial due to the different training strategies and data forms. To address this gap, we introduce SES-Adapter, a simple, efficient, and scalable adapter method for enhancing the representation learning of PLMs. SES-Adapter incorporates PLM embeddings with structural sequence embeddings to create structure-aware representations. We show that the proposed method is compatible with different PLM architectures and across diverse tasks. Extensive evaluations are conducted on 2 types of folding structures with notable quality differences, 9 state-of-the-art baselines, and 9 benchmark datasets across distinct downstream tasks. Results show that compared to vanilla PLMs, SES-Adapter improves downstream task performance by a maximum of 11% and an average of 3%, with significantly accelerated training speed by a maximum of 1034% and an average of 362%, the convergence rate is also improved by approximately 2 times. Moreover, positive optimization is observed even with low-quality predicted structures. The source code for SES-Adapter is available at https://github.com/tyang816/SES-Adapter.