# Learning Attention-based Representations from Multiple Patterns for Relation Prediction in Knowledge Graphs

arXiv: https://arxiv.org/abs/2206.04801 · HTML: https://mlsanity.com/paper/2206.04801/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [Inductive Relation Prediction by Subgraph Reasoning](https://mlsanity.com/paper/1911.06962/) (2019)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [SimplE Embedding for Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/1802.04868/) (2018)
- [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019)
- [End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completion](https://mlsanity.com/paper/1811.04441/) (2018)
- [Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs](https://mlsanity.com/paper/1906.01195/) (2019)
- [Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs](https://mlsanity.com/paper/1905.04914/) (2019)
- [Relational Message Passing for Knowledge Graph Completion](https://mlsanity.com/paper/2002.06757/) (2020)
- [Modeling Relation Paths for Representation Learning of Knowledge Bases](https://mlsanity.com/paper/1506.00379/) (2015)
- [Traversing Knowledge Graphs in Vector Space](https://mlsanity.com/paper/1506.01094/) (2015)
- [Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks](https://mlsanity.com/paper/1607.01426/) (2016)
- [Improving Knowledge Graph Embedding Using Simple Constraints](https://mlsanity.com/paper/1805.02408/) (2018)
- [Compositional Vector Space Models for Knowledge Base Completion](https://mlsanity.com/paper/1504.06662/) (2015)
- [Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval](https://mlsanity.com/paper/1805.07591/) (2018)
- [Relation Embedding with Dihedral Group in Knowledge Graph](https://mlsanity.com/paper/1906.00687/) (2019)
- [Dynamically Pruned Message Passing Networks for Large-Scale Knowledge Graph Reasoning](https://mlsanity.com/paper/1909.11334/) (2019)
- [Type-augmented Relation Prediction in Knowledge Graphs](https://mlsanity.com/paper/2009.07938/) (2020)
- [A Comparative Study of Distributional and Symbolic Paradigms for Relational Learning](https://mlsanity.com/paper/1806.11391/) (2018)
- [Association Rules Enhanced Knowledge Graph Attention Network](https://mlsanity.com/paper/2011.08431/) (2020)

## Abstract (arXiv)

Knowledge bases, and their representations in the form of knowledge graphs (KGs), are naturally incomplete. Since scientific and industrial applications have extensively adopted them, there is a high demand for solutions that complete their information. Several recent works tackle this challenge by learning embeddings for entities and relations, then employing them to predict new relations among the entities. Despite their aggrandizement, most of those methods focus only on the local neighbors of a relation to learn the embeddings. As a result, they may fail to capture the KGs' context information by neglecting long-term dependencies and the propagation of entities' semantics. In this manuscript, we propose AEMP (Attention-based Embeddings from Multiple Patterns), a novel model for learning contextualized representations by: (i) acquiring entities' context information through an attention-enhanced message-passing scheme, which captures the entities' local semantics while focusing on different aspects of their neighborhood; and (ii) capturing the semantic context, by leveraging the paths and their relationships between entities. Our empirical findings draw insights into how attention mechanisms can improve entities' context representation and how combining entities and semantic path contexts improves the general representation of entities and the relation predictions. Experimental results on several large and small knowledge graph benchmarks show that AEMP either outperforms or competes with state-of-the-art relation prediction methods.