# MoCoSA: Momentum Contrast for Knowledge Graph Completion with Structure-Augmented Pre-trained Language Models

arXiv: https://arxiv.org/abs/2308.08204 · HTML: https://mlsanity.com/paper/2308.08204/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities](https://mlsanity.com/paper/2305.13168/) (2023)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [Sequence-to-Sequence Knowledge Graph Completion and Question Answering](https://mlsanity.com/paper/2203.10321/) (2022)
- [Semi-Supervised Action Recognition with Temporal Contrastive Learning](https://mlsanity.com/paper/2102.02751/) (2021)
- [From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer](https://mlsanity.com/paper/2202.02113/) (2022)
- [Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study](https://mlsanity.com/paper/2003.08001/) (2020)
- [Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting](https://mlsanity.com/paper/2307.01709/) (2023)
- [Knowledge Is Flat: A Seq2Seq Generative Framework for Various Knowledge Graph Completion](https://mlsanity.com/paper/2209.07299/) (2022)
- [Language Models as Knowledge Embeddings](https://mlsanity.com/paper/2206.12617/) (2022)
- [Construction and Applications of Billion-Scale Pre-Trained Multimodal Business Knowledge Graph](https://mlsanity.com/paper/2209.15214/) (2022)

## Built on by (most cited first, 4 of 4)

- [Unsupervised Pretraining for Fact Verification by Language Model Distillation](https://mlsanity.com/paper/2309.16540/) (2023)
- [Knowledge Graph Completion using Structural and Textual Embeddings](https://mlsanity.com/paper/2404.16206/) (2024)
- [Harmonizing Human Insights and AI Precision: Hand in Hand for Advancing Knowledge Graph Task](https://mlsanity.com/paper/2405.09477/) (2024)
- [Enhancing Persona Classification in Dialogue Systems: A Graph Neural Network Approach](https://mlsanity.com/paper/2412.13283/) (2024)

## Abstract (arXiv)

Knowledge Graph Completion (KGC) aims to conduct reasoning on the facts within knowledge graphs and automatically infer missing links. Existing methods can mainly be categorized into structure-based or description-based. On the one hand, structure-based methods effectively represent relational facts in knowledge graphs using entity embeddings. However, they struggle with semantically rich real-world entities due to limited structural information and fail to generalize to unseen entities. On the other hand, description-based methods leverage pre-trained language models (PLMs) to understand textual information. They exhibit strong robustness towards unseen entities. However, they have difficulty with larger negative sampling and often lag behind structure-based methods. To address these issues, in this paper, we propose Momentum Contrast for knowledge graph completion with Structure-Augmented pre-trained language models (MoCoSA), which allows the PLM to perceive the structural information by the adaptable structure encoder. To improve learning efficiency, we proposed momentum hard negative and intra-relation negative sampling. Experimental results demonstrate that our approach achieves state-of-the-art performance in terms of mean reciprocal rank (MRR), with improvements of 2.5% on WN18RR and 21% on OpenBG500.