# Structure-Informed Protein Language Model

arXiv: https://arxiv.org/abs/2402.05856 · HTML: https://mlsanity.com/paper/2402.05856/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 11 of 11)

- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [Learning protein sequence embeddings using information from structure](https://mlsanity.com/paper/1902.08661/) (2019)
- [DeepSF: deep convolutional neural network for mapping protein sequences to folds](https://mlsanity.com/paper/1706.01010/) (2017)
- [Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling](https://mlsanity.com/paper/2301.06568/) (2023)
- [Structure-aware Protein Self-supervised Learning](https://mlsanity.com/paper/2204.04213/) (2022)
- [A Systematic Study of Joint Representation Learning on Protein Sequences and Structures](https://mlsanity.com/paper/2303.06275/) (2023)
- [Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Prediction](https://mlsanity.com/paper/2301.12068/) (2023)

## Built on by (most cited first, 6 of 6)

- [Simple, Efficient and Scalable Structure-aware Adapter Boosts Protein Language Models](https://mlsanity.com/paper/2404.14850/) (2024)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [SeqProFT: Sequence-only Protein Property Prediction with LoRA Finetuning](https://mlsanity.com/paper/2411.11530/) (2024)
- [Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference](https://mlsanity.com/paper/2602.20449/) (2026)
- [Structure-Aligned Protein Language Model](https://mlsanity.com/paper/2505.16896/) (2025)
- [LC-SEPLM: long-range contact-supervised adaptation for sequence-only protein representation learning](https://mlsanity.com/paper/2607.22777/) (2026)

## Abstract (arXiv)

Protein language models are a powerful tool for learning protein representations through pre-training on vast protein sequence datasets. However, traditional protein language models lack explicit structural supervision, despite its relevance to protein function. To address this issue, we introduce the integration of remote homology detection to distill structural information into protein language models without requiring explicit protein structures as input. We evaluate the impact of this structure-informed training on downstream protein function prediction tasks. Experimental results reveal consistent improvements in function annotation accuracy for EC number and GO term prediction. Performance on mutant datasets, however, varies based on the relationship between targeted properties and protein structures. This underscores the importance of considering this relationship when applying structure-aware training to protein function prediction tasks. Code and model weights are available at https://github.com/DeepGraphLearning/esm-s.