# Structure-aware Protein Self-supervised Learning

arXiv: https://arxiv.org/abs/2204.04213 · HTML: https://mlsanity.com/paper/2204.04213/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 16 of 16)

- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [Multi-Scale Representation Learning on Proteins](https://mlsanity.com/paper/2204.02337/) (2022)
- [Learning protein sequence embeddings using information from structure](https://mlsanity.com/paper/1902.08661/) (2019)
- [DeepSF: deep convolutional neural network for mapping protein sequences to folds](https://mlsanity.com/paper/1706.01010/) (2017)
- [End-to-End Learning on 3D Protein Structure for Interface Prediction](https://mlsanity.com/paper/1807.01297/) (2018)
- [Generalized Data Weighting via Class-level Gradient Manipulation](https://mlsanity.com/paper/2111.00056/) (2021)
- [Convolutional LSTM Networks for Subcellular Localization of Proteins](https://mlsanity.com/paper/1503.01919/) (2015)

## Built on by (most cited first, 30 of 33)

- [Low-Light Video Enhancement with Synthetic Event Guidance](https://mlsanity.com/paper/2208.11014/) (2022)
- [Gradient-based Bi-level Optimization for Deep Learning: A Survey](https://mlsanity.com/paper/2207.11719/) (2022)
- [Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences](https://mlsanity.com/paper/2306.03111/) (2023)
- [MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding](https://mlsanity.com/paper/2402.14391/) (2024)
- [Protein Language Models and Structure Prediction: Connection and Progression](https://mlsanity.com/paper/2211.16742/) (2022)
- [ParetoFlow: Guided Flows in Multi-Objective Optimization](https://mlsanity.com/paper/2412.03718/) (2024)
- [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025)
- [Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Prediction](https://mlsanity.com/paper/2301.12068/) (2023)
- [A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design](https://mlsanity.com/paper/2311.16126/) (2023)
- [A Survey on Protein Representation Learning: Retrospect and Prospect](https://mlsanity.com/paper/2301.00813/) (2022)
- [Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy](https://mlsanity.com/paper/2302.10888/) (2023)
- [Design Editing for Offline Model-based Optimization](https://mlsanity.com/paper/2405.13964/) (2024)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [Structure-Informed Protein Language Model](https://mlsanity.com/paper/2402.05856/) (2024)
- [Pre-Training on Large-Scale Generated Docking Conformations with HelixDock to Unlock the Potential of Protein-ligand Structure Prediction Models](https://mlsanity.com/paper/2310.13913/) (2023)
- [Using pretrained graph neural networks with token mixers as geometric featurizers for conformational dynamics](https://mlsanity.com/paper/2409.19838/) (2024)
- [On Pre-trained Language Models for Antibody](https://mlsanity.com/paper/2301.12112/) (2023)
- [3DMolFormer: A Dual-channel Framework for Structure-based Drug Discovery](https://mlsanity.com/paper/2502.05107/) (2025)
- [Bi-Level Optimization for Self-Supervised AI-Generated Face Detection](https://mlsanity.com/paper/2507.22824/) (2025)
- [Protein Representation Learning via Knowledge Enhanced Primary Structure Modeling](https://mlsanity.com/paper/2301.13154/) (2023)
- [MeToken: Uniform Micro-environment Token Boosts Post-Translational Modification Prediction](https://mlsanity.com/paper/2411.01856/) (2024)
- [SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation](https://mlsanity.com/paper/2410.24022/) (2024)
- [S2ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning](https://mlsanity.com/paper/2411.15215/) (2024)
- [Graph Representation Learning for Interactive Biomolecule Systems](https://mlsanity.com/paper/2304.02656/) (2023)
- [AffinityFlow: Guided Flows for Antibody Affinity Maturation](https://mlsanity.com/paper/2502.10365/) (2025)
- [Structure-Aligned Protein Language Model](https://mlsanity.com/paper/2505.16896/) (2025)
- [ProtIR: Iterative Refinement between Retrievers and Predictors for Protein Function Annotation](https://mlsanity.com/paper/2402.07955/) (2024)
- [NaNa and MiGu: Semantic Data Augmentation Techniques to Enhance Protein Classification in Graph Neural Networks](https://mlsanity.com/paper/2403.14736/) (2024)
- [Protein Representation Learning by Capturing Protein Sequence-Structure-Function Relationship](https://mlsanity.com/paper/2405.06663/) (2024)
- [Advanced atom-level representations for protein flexibility prediction utilizing graph neural networks](https://mlsanity.com/paper/2408.12519/) (2024)

## Abstract (arXiv)

Protein representation learning methods have shown great potential to yield useful representation for many downstream tasks, especially on protein classification. Moreover, a few recent studies have shown great promise in addressing insufficient labels of proteins with self-supervised learning methods. However, existing protein language models are usually pretrained on protein sequences without considering the important protein structural information. To this end, we propose a novel structure-aware protein self-supervised learning method to effectively capture structural information of proteins. In particular, a well-designed graph neural network (GNN) model is pretrained to preserve the protein structural information with self-supervised tasks from a pairwise residue distance perspective and a dihedral angle perspective, respectively. Furthermore, we propose to leverage the available protein language model pretrained on protein sequences to enhance the self-supervised learning. Specifically, we identify the relation between the sequential information in the protein language model and the structural information in the specially designed GNN model via a novel pseudo bi-level optimization scheme. Experiments on several supervised downstream tasks verify the effectiveness of our proposed method.The code of the proposed method is available in urlhttps://github.com/GGchen1997/STEPSBioinformatics.