# Bidirectional Hierarchical Protein Multi-Modal Representation Learning

arXiv: https://arxiv.org/abs/2504.04770 · HTML: https://mlsanity.com/paper/2504.04770/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 9 of 9)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2103.12978/) (2021)
- [Protein Representation Learning by Geometric Structure Pretraining](https://mlsanity.com/paper/2203.06125/) (2022)
- [ComENet: Towards Complete and Efficient Message Passing for 3D Molecular Graphs](https://mlsanity.com/paper/2206.08515/) (2022)
- [Learning Hierarchical Protein Representations via Complete 3D Graph Networks](https://mlsanity.com/paper/2207.12600/) (2022)
- [Multi-Scale Representation Learning on Proteins](https://mlsanity.com/paper/2204.02337/) (2022)
- [A Systematic Study of Joint Representation Learning on Protein Sequences and Structures](https://mlsanity.com/paper/2303.06275/) (2023)
- [Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks](https://mlsanity.com/paper/2212.03447/) (2022)
- [Binding Affinity Prediction: From Conventional to Machine Learning-Based Approaches](https://mlsanity.com/paper/2410.00709/) (2024)

## Built on by (most cited first, 1 of 1)

- [A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design](https://mlsanity.com/paper/2507.13646/) (2025)

## Abstract (arXiv)

Protein representation learning is critical for numerous biological tasks. Recently, large transformer-based protein language models (pLMs) pretrained on large scale protein sequences have demonstrated significant success in sequence-based tasks. However, pLMs lack structural context. Conversely, graph neural networks (GNNs) designed to leverage 3D structural information have shown promising generalization in protein-related prediction tasks, but their effectiveness is often constrained by the scarcity of labeled structural data. Recognizing that sequence and structural representations are complementary perspectives of the same protein entity, we propose a multimodal bidirectional hierarchical fusion framework to effectively merge these modalities. Our framework employs attention and gating mechanisms to enable effective interaction between pLMs-generated sequential representations and GNN-extracted structural features, improving information exchange and enhancement across layers of the neural network. This bidirectional and hierarchical (Bi-Hierarchical) fusion approach leverages the strengths of both modalities to capture richer and more comprehensive protein representations. Based on the framework, we further introduce local Bi-Hierarchical Fusion with gating and global Bi-Hierarchical Fusion with multihead self-attention approaches. Our method demonstrates consistent improvements over strong baselines and existing fusion techniques in a variety of protein representation learning benchmarks, including enzyme EC classification, model quality assessment, protein-ligand binding affinity prediction, protein-protein binding site prediction, and B cell epitopes prediction. Our method establishes a new state-of-the-art for multimodal protein representation learning, emphasizing the efficacy of Bi-Hierarchical Fusion in bridging sequence and structural modalities.