# Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering

arXiv: https://arxiv.org/abs/2210.10462 · HTML: https://mlsanity.com/paper/2210.10462/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 47 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [Graph Representation Learning via Graphical Mutual Information Maximization](https://mlsanity.com/paper/2002.01169/) (2020)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representations](https://mlsanity.com/paper/1911.07979/) (2019)
- [Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction](https://mlsanity.com/paper/2111.00064/) (2021)
- [Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive](https://mlsanity.com/paper/2105.07342/) (2021)
- [Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning](https://mlsanity.com/paper/2105.09111/) (2021)
- [Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels](https://mlsanity.com/paper/1902.11038/) (2019)
- [InfoGCL: Information-Aware Graph Contrastive Learning](https://mlsanity.com/paper/2110.15438/) (2021)
- [Unifying Graph Convolutional Neural Networks and Label Propagation](https://mlsanity.com/paper/2002.06755/) (2020)
- [Unsupervised Attributed Multiplex Network Embedding](https://mlsanity.com/paper/1911.06750/) (2019)
- [HDMI: High-order Deep Multiplex Infomax](https://mlsanity.com/paper/2102.07810/) (2021)
- [Interpretable and Efficient Heterogeneous Graph Convolutional Network](https://mlsanity.com/paper/2005.13183/) (2020)
- [Heterogeneous Deep Graph Infomax](https://mlsanity.com/paper/1911.08538/) (2019)

## Built on by (most cited first, 30 of 30)

- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [Hard Sample Aware Network for Contrastive Deep Graph Clustering](https://mlsanity.com/paper/2212.08665/) (2022)
- [HGPROMPT: Bridging Homogeneous and Heterogeneous Graphs for Few-shot Prompt Learning](https://mlsanity.com/paper/2312.01878/) (2023)
- [HiGPT: Heterogeneous Graph Language Model](https://mlsanity.com/paper/2402.16024/) (2024)
- [HetGPT: Harnessing the Power of Prompt Tuning in Pre-Trained Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2310.15318/) (2023)
- [Learn from Global Correlations: Enhancing Evolutionary Algorithm via Spectral GNN](https://mlsanity.com/paper/2412.17629/) (2024)
- [GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuning](https://mlsanity.com/paper/2511.05592/) (2025)
- [End-to-end Learnable Clustering for Intent Learning in Recommendation](https://mlsanity.com/paper/2401.05975/) (2024)
- [Semantic-aware Node Synthesis for Imbalanced Heterogeneous Information Networks](https://mlsanity.com/paper/2302.14061/) (2023)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [RAG-GFM: Overcoming In-Memory Bottlenecks in Graph Foundation Models via Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.15124/) (2026)
- [Pseudo Contrastive Learning for Graph-based Semi-supervised Learning](https://mlsanity.com/paper/2302.09532/) (2023)
- [Challenges in Pre-Training Graph Neural Networks for Context-Based Fake News Detection: An Evaluation of Current Strategies and Resource Limitations](https://mlsanity.com/paper/2402.18179/) (2024)
- [HHGT: Hierarchical Heterogeneous Graph Transformer for Heterogeneous Graph Representation Learning](https://mlsanity.com/paper/2407.13158/) (2024)
- [HG-Adapter: Improving Pre-Trained Heterogeneous Graph Neural Networks with Dual Adapters](https://mlsanity.com/paper/2411.01155/) (2024)
- [HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networks](https://mlsanity.com/paper/2501.12857/) (2025)
- [Enhancing Homophily-Heterophily Separation: Relation-Aware Learning in Heterogeneous Graphs](https://mlsanity.com/paper/2506.20980/) (2025)
- [Towards Efficient LLM-aware Heterogeneous Graph Learning](https://mlsanity.com/paper/2511.17923/) (2025)
- [Retrieving Minimal and Sufficient Reasoning Subgraphs with Graph Foundation Models for Path-aware GraphRAG](https://mlsanity.com/paper/2603.07179/) (2026)
- [Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networks](https://mlsanity.com/paper/2311.11368/) (2023)
- [Transfer Learning Under High-Dimensional Graph Convolutional Regression Model for Node Classification](https://mlsanity.com/paper/2405.16672/) (2024)
- [Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks](https://mlsanity.com/paper/2505.07895/) (2025)
- [Empowering Heterogeneous Graph Foundation Models via Decoupled Relation Alignment](https://mlsanity.com/paper/2605.00731/) (2026)
- [Incorporating Attributes and Multi-Scale Structures for Heterogeneous Graph Contrastive Learning](https://mlsanity.com/paper/2503.13911/) (2025)
- [Masked Language Models are Good Heterogeneous Graph Generalizers](https://mlsanity.com/paper/2506.06157/) (2025)
- [GFlowNets for Learning Better Drug-Drug Interaction Representations](https://mlsanity.com/paper/2508.06576/) (2025)
- [Enhanced Pre-training of Graph Neural Networks for Million-Scale Heterogeneous Graphs](https://mlsanity.com/paper/2510.12401/) (2025)
- [Mixing Configurations for Downstream Prediction](https://mlsanity.com/paper/2510.19248/) (2025)
- [THeGAU: Type-Aware Heterogeneous Graph Autoencoder and Augmentation](https://mlsanity.com/paper/2512.10589/) (2025)
- [FOCAL-Attention for Heterogeneous Multi-Label Prediction](https://mlsanity.com/paper/2604.19171/) (2026)

## Abstract (arXiv)

Recent self-supervised pre-training methods on Heterogeneous Information Networks (HINs) have shown promising competitiveness over traditional semi-supervised Heterogeneous Graph Neural Networks (HGNNs). Unfortunately, their performance heavily depends on careful customization of various strategies for generating high-quality positive examples and negative examples, which notably limits their flexibility and generalization ability. In this work, we present SHGP, a novel Self-supervised Heterogeneous Graph Pre-training approach, which does not need to generate any positive examples or negative examples. It consists of two modules that share the same attention-aggregation scheme. In each iteration, the Att-LPA module produces pseudo-labels through structural clustering, which serve as the self-supervision signals to guide the Att-HGNN module to learn object embeddings and attention coefficients. The two modules can effectively utilize and enhance each other, promoting the model to learn discriminative embeddings. Extensive experiments on four real-world datasets demonstrate the superior effectiveness of SHGP against state-of-the-art unsupervised baselines and even semi-supervised baselines. We release our source code at: https://github.com/kepsail/SHGP.