# Contrastive Representation Learning for 3D Protein Structures

arXiv: https://arxiv.org/abs/2205.15675 · HTML: https://mlsanity.com/paper/2205.15675/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 72 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Learning from Protein Structure with Geometric Vector Perceptrons](https://mlsanity.com/paper/2009.01411/) (2020)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [DeepDTA: Deep Drug-Target Binding Affinity Prediction](https://mlsanity.com/paper/1801.10193/) (2018)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [ATOM3D: Tasks On Molecules in Three Dimensions](https://mlsanity.com/paper/2012.04035/) (2020)
- [Protein-Ligand Scoring with Convolutional Neural Networks](https://mlsanity.com/paper/1612.02751/) (2016)
- [Edge Contraction Pooling for Graph Neural Networks](https://mlsanity.com/paper/1905.10990/) (2019)
- [Multi-Scale Representation Learning on Proteins](https://mlsanity.com/paper/2204.02337/) (2022)
- [Learning protein sequence embeddings using information from structure](https://mlsanity.com/paper/1902.08661/) (2019)
- [DeepSF: deep convolutional neural network for mapping protein sequences to folds](https://mlsanity.com/paper/1706.01010/) (2017)
- [End-to-End Learning on 3D Protein Structure for Interface Prediction](https://mlsanity.com/paper/1807.01297/) (2018)
- [Is Transfer Learning Necessary for Protein Landscape Prediction?](https://mlsanity.com/paper/2011.03443/) (2020)
- [Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information](https://mlsanity.com/paper/1912.05625/) (2019)

## Built on by (most cited first, 30 of 30)

- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [Learning Hierarchical Protein Representations via Complete 3D Graph Networks](https://mlsanity.com/paper/2207.12600/) (2022)
- [DiffPack: A Torsional Diffusion Model for Autoregressive Protein Side-Chain Packing](https://mlsanity.com/paper/2306.01794/) (2023)
- [Generalist Equivariant Transformer Towards 3D Molecular Interaction Learning](https://mlsanity.com/paper/2306.01474/) (2023)
- [A Systematic Study of Joint Representation Learning on Protein Sequences and Structures](https://mlsanity.com/paper/2303.06275/) (2023)
- [Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks](https://mlsanity.com/paper/2212.03447/) (2022)
- [De novo protein design using geometric vector field networks](https://mlsanity.com/paper/2310.11802/) (2023)
- [Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Prediction](https://mlsanity.com/paper/2301.12068/) (2023)
- [A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design](https://mlsanity.com/paper/2311.16126/) (2023)
- [Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy](https://mlsanity.com/paper/2302.10888/) (2023)
- [PSC-CPI: Multi-Scale Protein Sequence-Structure Contrasting for Efficient and Generalizable Compound-Protein Interaction Prediction](https://mlsanity.com/paper/2402.08198/) (2024)
- [Instructor-inspired Machine Learning for Robust Molecular Property Prediction](https://mlsanity.com/paper/2304.03906/) (2023)
- [ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling](https://mlsanity.com/paper/2403.12995/) (2024)
- [Clustering for Protein Representation Learning](https://mlsanity.com/paper/2404.00254/) (2024)
- [Using pretrained graph neural networks with token mixers as geometric featurizers for conformational dynamics](https://mlsanity.com/paper/2409.19838/) (2024)
- [GLProtein: Global-and-Local Structure Aware Protein Representation Learning](https://mlsanity.com/paper/2506.06294/) (2025)
- [CCPL: Cross-modal Contrastive Protein Learning](https://mlsanity.com/paper/2303.11783/) (2023)
- [EurNet: Efficient Multi-Range Relational Modeling of Spatial Multi-Relational Data](https://mlsanity.com/paper/2211.12941/) (2022)
- [Advances in Protein Representation Learning: Methods, Applications, and Future Directions](https://mlsanity.com/paper/2503.16659/) (2025)
- [SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation](https://mlsanity.com/paper/2410.24022/) (2024)
- [Graph Representation Learning for Interactive Biomolecule Systems](https://mlsanity.com/paper/2304.02656/) (2023)
- [Protein Representation Learning by Capturing Protein Sequence-Structure-Function Relationship](https://mlsanity.com/paper/2405.06663/) (2024)
- [Autoregressive Enzyme Function Prediction with Multi-scale Multi-modality Fusion](https://mlsanity.com/paper/2408.06391/) (2024)
- [Hashing for Protein Structure Similarity Search](https://mlsanity.com/paper/2411.08286/) (2024)
- [A Novel Framework for Multi-Modal Protein Representation Learning](https://mlsanity.com/paper/2510.23273/) (2025)
- [MMPG: MoE-based Adaptive Multi-Perspective Graph Fusion for Protein Representation Learning](https://mlsanity.com/paper/2601.10157/) (2026)
- [Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles](https://mlsanity.com/paper/2603.02406/) (2026)
- [Protein Representation Learning with Secondary-Structure and Energy-Filtered Hydrogen-Bond Graphs](https://mlsanity.com/paper/2606.19374/) (2026)
- [Enhancing Protein Representation Learning via Manifold Restore Mixing](https://mlsanity.com/paper/2606.22307/) (2026)
- [Structure-aware Reinforcement Learning for Protein Directed Evolution](https://mlsanity.com/paper/2609.39048/) (2026)

## Abstract (arXiv)

Learning from 3D protein structures has gained wide interest in protein modeling and structural bioinformatics. Unfortunately, the number of available structures is orders of magnitude lower than the training data sizes commonly used in computer vision and machine learning. Moreover, this number is reduced even further, when only annotated protein structures can be considered, making the training of existing models difficult and prone to over-fitting. To address this challenge, we introduce a new representation learning framework for 3D protein structures. Our framework uses unsupervised contrastive learning to learn meaningful representations of protein structures, making use of proteins from the Protein Data Bank. We show, how these representations can be used to solve a large variety of tasks, such as protein function prediction, protein fold classification, structural similarity prediction, and protein-ligand binding affinity prediction. Moreover, we show how fine-tuned networks, pre-trained with our algorithm, lead to significantly improved task performance, achieving new state-of-the-art results in many tasks.