# Kronecker Decomposition for Knowledge Graph Embeddings

arXiv: https://arxiv.org/abs/2205.06560 · HTML: https://mlsanity.com/paper/2205.06560/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 72 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications](https://mlsanity.com/paper/1709.07604/) (2017)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Holographic Embeddings of Knowledge Graphs](https://mlsanity.com/paper/1510.04935/) (2015)
- [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [Canonical Tensor Decomposition for Knowledge Base Completion](https://mlsanity.com/paper/1806.07297/) (2018)
- [A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network](https://mlsanity.com/paper/1712.02121/) (2017)
- [ALP-KD: Attention-Based Layer Projection for Knowledge Distillation](https://mlsanity.com/paper/2012.14022/) (2020)
- [Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters](https://mlsanity.com/paper/2102.08597/) (2021)
- [Hypernetwork Knowledge Graph Embeddings](https://mlsanity.com/paper/1808.07018/) (2018)
- [Kronecker Decomposition for GPT Compression](https://mlsanity.com/paper/2110.08152/) (2021)
- [MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation](https://mlsanity.com/paper/2105.05912/) (2021)

## Built on by (most cited first, 2 of 2)

- [Conformal Credal Self-Supervised Learning](https://mlsanity.com/paper/2205.15239/) (2022)
- [Hardware-agnostic Computation for Large-scale Knowledge Graph Embeddings](https://mlsanity.com/paper/2207.08544/) (2022)

## Abstract (arXiv)

Knowledge graph embedding research has mainly focused on learning continuous representations of entities and relations tailored towards the link prediction problem. Recent results indicate an ever increasing predictive ability of current approaches on benchmark datasets. However, this effectiveness often comes with the cost of over-parameterization and increased computationally complexity. The former induces extensive hyperparameter optimization to mitigate malicious overfitting. The latter magnifies the importance of winning the hardware lottery. Here, we investigate a remedy for the first problem. We propose a technique based on Kronecker decomposition to reduce the number of parameters in a knowledge graph embedding model, while retaining its expressiveness. Through Kronecker decomposition, large embedding matrices are split into smaller embedding matrices during the training process. Hence, embeddings of knowledge graphs are not plainly retrieved but reconstructed on the fly. The decomposition ensures that elementwise interactions between three embedding vectors are extended with interactions within each embedding vector. This implicitly reduces redundancy in embedding vectors and encourages feature reuse. To quantify the impact of applying Kronecker decomposition on embedding matrices, we conduct a series of experiments on benchmark datasets. Our experiments suggest that applying Kronecker decomposition on embedding matrices leads to an improved parameter efficiency on all benchmark datasets. Moreover, empirical evidence suggests that reconstructed embeddings entail robustness against noise in the input knowledge graph. To foster reproducible research, we provide an open-source implementation of our approach, including training and evaluation scripts as well as pre-trained models in our knowledge graph embedding framework (https://github.com/dice-group/dice-embeddings).