# Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks

arXiv: https://arxiv.org/abs/2007.11192 · HTML: https://mlsanity.com/paper/2007.11192/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 17 of 17)

- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Self-supervised Learning on Graphs: Contrastive, Generative,or Predictive](https://mlsanity.com/paper/2105.07342/) (2021)
- [Graph Prompt Learning: A Comprehensive Survey and Beyond](https://mlsanity.com/paper/2311.16534/) (2023)
- [Graph Neural Networks: Methods, Applications, and Opportunities](https://mlsanity.com/paper/2108.10733/) (2021)
- [Detecting Communities from Heterogeneous Graphs: A Context Path-based Graph Neural Network Model](https://mlsanity.com/paper/2109.02058/) (2021)
- [Universal Link Predictor By In-Context Learning on Graphs](https://mlsanity.com/paper/2402.07738/) (2024)
- [SMiLE: Schema-augmented Multi-level Contrastive Learning for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2210.04870/) (2022)
- [A Graph is Worth K Words: Euclideanizing Graph using Pure Transformer](https://mlsanity.com/paper/2402.02464/) (2024)
- [Message Intercommunication for Inductive Relation Reasoning](https://mlsanity.com/paper/2305.14074/) (2023)
- [Link Prediction with Contextualized Self-Supervision](https://mlsanity.com/paper/2201.10069/) (2022)
- [Few-Shot Semantic Relation Prediction across Heterogeneous Graphs](https://mlsanity.com/paper/2207.05068/) (2022)
- [Local Structure-aware Graph Contrastive Representation Learning](https://mlsanity.com/paper/2308.03271/) (2023)
- [Position-Aware Subgraph Neural Networks with Data-Efficient Learning](https://mlsanity.com/paper/2211.00572/) (2022)
- [Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings](https://mlsanity.com/paper/2305.09316/) (2023)
- [SCGG: A Deep Structure-Conditioned Graph Generative Model](https://mlsanity.com/paper/2209.09681/) (2022)
- [Self-supervised Learning for Heterogeneous Graph via Structure Information based on Metapath](https://mlsanity.com/paper/2209.04218/) (2022)
- [NumCoKE: Ordinal-Aware Numerical Reasoning over Knowledge Graphs with Mixture-of-Experts and Contrastive Learning](https://mlsanity.com/paper/2411.12950/) (2024)

## Abstract (arXiv)

Representation learning methods for heterogeneous networks produce a low-dimensional vector embedding for each node that is typically fixed for all tasks involving the node. Many of the existing methods focus on obtaining a static vector representation for a node in a way that is agnostic to the downstream application where it is being used. In practice, however, downstream tasks such as link prediction require specific contextual information that can be extracted from the subgraphs related to the nodes provided as input to the task. To tackle this challenge, we develop SLiCE, a framework bridging static representation learning methods using global information from the entire graph with localized attention driven mechanisms to learn contextual node representations. We first pre-train our model in a self-supervised manner by introducing higher-order semantic associations and masking nodes, and then fine-tune our model for a specific link prediction task. Instead of training node representations by aggregating information from all semantic neighbors connected via metapaths, we automatically learn the composition of different metapaths that characterize the context for a specific task without the need for any pre-defined metapaths. SLiCE significantly outperforms both static and contextual embedding learning methods on several publicly available benchmark network datasets. We also interpret the semantic association matrix and provide its utility and relevance in making successful link predictions between heterogeneous nodes in the network.