# Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis

arXiv: https://arxiv.org/abs/2104.04676 · HTML: https://mlsanity.com/paper/2104.04676/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 51 references that are in the map.
- First posted on arXiv 10 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning](https://mlsanity.com/paper/2002.05651/) (2020)
- [Low-Dimensional Hyperbolic Knowledge Graph Embeddings](https://mlsanity.com/paper/2005.00545/) (2020)
- [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019)
- [A Survey on Knowledge Graph-Based Recommender Systems](https://mlsanity.com/paper/2003.00911/) (2020)
- [End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completion](https://mlsanity.com/paper/1811.04441/) (2018)
- [Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings](https://mlsanity.com/paper/1704.07130/) (2017)
- [InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions](https://mlsanity.com/paper/1911.00219/) (2019)
- [Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding](https://mlsanity.com/paper/1911.04910/) (2019)
- [Measuring Social Bias in Knowledge Graph Embeddings](https://mlsanity.com/paper/1912.02761/) (2019)
- [SEEK: Segmented Embedding of Knowledge Graphs](https://mlsanity.com/paper/2005.00856/) (2020)
- [Towards Understanding the Importance of Noise in Training Neural Networks](https://mlsanity.com/paper/1909.03172/) (2019)
- [Fast Linear Model for Knowledge Graph Embeddings](https://mlsanity.com/paper/1710.10881/) (2017)

## Built on by (most cited first, 10 of 10)

- [Knowledge Graph Embedding: A Survey from the Perspective of Representation Spaces](https://mlsanity.com/paper/2211.03536/) (2022)
- [Multilingual Knowledge Graph Completion with Self-Supervised Adaptive Graph Alignment](https://mlsanity.com/paper/2203.14987/) (2022)
- [How Does Knowledge Graph Embedding Extrapolate to Unseen Data: A Semantic Evidence View](https://mlsanity.com/paper/2109.11800/) (2021)
- [Sem@K: Is my knowledge graph embedding model semantic-aware?](https://mlsanity.com/paper/2301.05601/) (2023)
- [Swift and Sure: Hardness-aware Contrastive Learning for Low-dimensional Knowledge Graph Embeddings](https://mlsanity.com/paper/2201.00565/) (2022)
- [Data Collaboration Analysis with Orthonormal Basis Selection and Alignment](https://mlsanity.com/paper/2403.02780/) (2024)
- [Is There More Pattern in Knowledge Graph? Exploring Proximity Pattern for Knowledge Graph Embedding](https://mlsanity.com/paper/2110.00720/) (2021)
- [Assessing the Effects of Hyperparameters on Knowledge Graph Embedding Quality](https://mlsanity.com/paper/2207.00473/) (2022)
- [Can Persistent Homology provide an efficient alternative for Evaluation of Knowledge Graph Completion Methods?](https://mlsanity.com/paper/2301.12929/) (2023)
- [Node-based Knowledge Graph Contrastive Learning for Medical Relationship Prediction](https://mlsanity.com/paper/2310.10138/) (2023)

## Abstract (arXiv)

Knowledge Graph Embeddings (KGEs) have been intensively explored in recent years due to their promise for a wide range of applications. However, existing studies focus on improving the final model performance without acknowledging the computational cost of the proposed approaches, in terms of execution time and environmental impact. This paper proposes a simple yet effective KGE framework which can reduce the training time and carbon footprint by orders of magnitudes compared with state-of-the-art approaches, while producing competitive performance. We highlight three technical innovations: full batch learning via relational matrices, closed-form Orthogonal Procrustes Analysis for KGEs, and non-negative-sampling training. In addition, as the first KGE method whose entity embeddings also store full relation information, our trained models encode rich semantics and are highly interpretable. Comprehensive experiments and ablation studies involving 13 strong baselines and two standard datasets verify the effectiveness and efficiency of our algorithm.