# SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models

arXiv: https://arxiv.org/abs/2203.02167 · HTML: https://mlsanity.com/paper/2203.02167/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 4 Mar 2022.

## Builds on (most cited first, 30 of 45)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)

## Built on by (most cited first, 30 of 91)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities](https://mlsanity.com/paper/2305.13168/) (2023)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure](https://mlsanity.com/paper/2211.10738/) (2022)
- [Exploring Large Language Models for Knowledge Graph Completion](https://mlsanity.com/paper/2308.13916/) (2023)
- [Guideline Learning for In-context Information Extraction](https://mlsanity.com/paper/2310.05066/) (2023)
- [Editing Language Model-based Knowledge Graph Embeddings](https://mlsanity.com/paper/2301.10405/) (2023)
- [KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge](https://mlsanity.com/paper/2405.16412/) (2024)
- [Research Trends for the Interplay between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2406.08223/) (2024)
- [Generalizing to Unseen Elements: A Survey on Knowledge Extrapolation for Knowledge Graphs](https://mlsanity.com/paper/2302.01859/) (2023)
- [Enhancing Text-based Knowledge Graph Completion with Zero-Shot Large Language Models: A Focus on Semantic Enhancement](https://mlsanity.com/paper/2310.08279/) (2023)
- [Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?](https://mlsanity.com/paper/2212.10784/) (2022)
- [Multimedia Generative Script Learning for Task Planning](https://mlsanity.com/paper/2208.12306/) (2022)
- [SMiLE: Schema-augmented Multi-level Contrastive Learning for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2210.04870/) (2022)
- [KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion](https://mlsanity.com/paper/2208.07622/) (2022)
- [Knowledge Graph Embedding: An Overview](https://mlsanity.com/paper/2309.12501/) (2023)
- [On the Evolution of Knowledge Graphs: A Survey and Perspective](https://mlsanity.com/paper/2310.04835/) (2023)
- [GreenKGC: A Lightweight Knowledge Graph Completion Method](https://mlsanity.com/paper/2208.09137/) (2022)
- [Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Prediction](https://mlsanity.com/paper/2304.04717/) (2023)
- [Friendly Neighbors: Contextualized Sequence-to-Sequence Link Prediction](https://mlsanity.com/paper/2305.13059/) (2023)
- [Negative Sampling in Knowledge Graph Representation Learning: A Review](https://mlsanity.com/paper/2402.19195/) (2024)
- [Relphormer: Relational Graph Transformer for Knowledge Graph Representations](https://mlsanity.com/paper/2205.10852/) (2022)
- [mOKB6: A Multilingual Open Knowledge Base Completion Benchmark](https://mlsanity.com/paper/2211.06959/) (2022)
- [Large Language Model Enhanced Knowledge Representation Learning: A Survey](https://mlsanity.com/paper/2407.00936/) (2024)
- [NeuSTIP: A Novel Neuro-Symbolic Model for Link and Time Prediction in Temporal Knowledge Graphs](https://mlsanity.com/paper/2305.11301/) (2023)
- [CompanyKG: A Large-Scale Heterogeneous Graph for Company Similarity Quantification](https://mlsanity.com/paper/2306.10649/) (2023)
- [KERMIT: Knowledge Graph Completion of Enhanced Relation Modeling with Inverse Transformation](https://mlsanity.com/paper/2309.14770/) (2023)
- [Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph](https://mlsanity.com/paper/2411.19064/) (2024)
- [LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings](https://mlsanity.com/paper/2210.00305/) (2022)
- [NNKGC: Improving Knowledge Graph Completion with Node Neighborhoods](https://mlsanity.com/paper/2302.06132/) (2023)

## Abstract (arXiv)

Knowledge graph completion (KGC) aims to reason over known facts and infer the missing links. Text-based methods such as KGBERT (Yao et al., 2019) learn entity representations from natural language descriptions, and have the potential for inductive KGC. However, the performance of text-based methods still largely lag behind graph embedding-based methods like TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019b). In this paper, we identify that the key issue is efficient contrastive learning. To improve the learning efficiency, we introduce three types of negatives: in-batch negatives, pre-batch negatives, and self-negatives which act as a simple form of hard negatives. Combined with InfoNCE loss, our proposed model SimKGC can substantially outperform embedding-based methods on several benchmark datasets. In terms of mean reciprocal rank (MRR), we advance the state-of-the-art by +19% on WN18RR, +6.8% on the Wikidata5M transductive setting, and +22% on the Wikidata5M inductive setting. Thorough analyses are conducted to gain insights into each component. Our code is available at https://github.com/intfloat/SimKGC .