# Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training

arXiv: https://arxiv.org/abs/2112.04087 · HTML: https://mlsanity.com/paper/2112.04087/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [SimplE Embedding for Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/1802.04868/) (2018)
- [Corpus-level Fine-grained Entity Typing Using Contextual Information](https://mlsanity.com/paper/1606.07901/) (2016)

## Built on by (most cited first, 2 of 2)

- [Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks](https://mlsanity.com/paper/2405.13085/) (2024)
- [KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models](https://mlsanity.com/paper/2506.03576/) (2025)

## Abstract (arXiv)

Representation learning models for Knowledge Graphs (KG) have proven to be effective in encoding structural information and performing reasoning over KGs. In this paper, we propose a novel pre-training-then-fine-tuning framework for knowledge graph representation learning, in which a KG model is firstly pre-trained with triple classification task, followed by discriminative fine-tuning on specific downstream tasks such as entity type prediction and entity alignment. Drawing on the general ideas of learning deep contextualized word representations in typical pre-trained language models, we propose SCoP to learn pre-trained KG representations with structural and contextual triples of the target triple encoded. Experimental results demonstrate that fine-tuning SCoP not only outperforms results of baselines on a portfolio of downstream tasks but also avoids tedious task-specific model design and parameter training.