# Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding

arXiv: https://arxiv.org/abs/2106.07250 · HTML: https://mlsanity.com/paper/2106.07250/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings](https://mlsanity.com/paper/2007.14175/) (2020)
- [Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling](https://mlsanity.com/paper/1906.07241/) (2019)
- [Understanding Negative Sampling in Graph Representation Learning](https://mlsanity.com/paper/2005.09863/) (2020)
- [Pre-Training Transformers as Energy-Based Cloze Models](https://mlsanity.com/paper/2012.08561/) (2020)
- [Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing](https://mlsanity.com/paper/2005.00820/) (2020)
- [Knowledge Base Completion: Baselines Strike Back](https://mlsanity.com/paper/1705.10744/) (2017)
- [Notes on Noise Contrastive Estimation and Negative Sampling](https://mlsanity.com/paper/1410.8251/) (2014)
- [Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing](https://mlsanity.com/paper/1802.05930/) (2018)
- [A Simple Language Model based on PMI Matrix Approximations](https://mlsanity.com/paper/1707.05266/) (2017)

## Built on by (most cited first, 7 of 7)

- [Comprehensive Analysis of Negative Sampling in Knowledge Graph Representation Learning](https://mlsanity.com/paper/2206.10140/) (2022)
- [Towards Artwork Explanation in Large-scale Vision Language Models](https://mlsanity.com/paper/2403.00068/) (2024)
- [Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models](https://mlsanity.com/paper/2409.01584/) (2024)
- [Subsampling for Knowledge Graph Embedding Explained](https://mlsanity.com/paper/2209.12801/) (2022)
- [Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?](https://mlsanity.com/paper/2311.09109/) (2023)
- [Model-based Subsampling for Knowledge Graph Completion](https://mlsanity.com/paper/2309.09296/) (2023)
- [Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding](https://mlsanity.com/paper/2407.04251/) (2024)

## Abstract (arXiv)

In knowledge graph embedding, the theoretical relationship between the softmax cross-entropy and negative sampling loss functions has not been investigated. This makes it difficult to fairly compare the results of the two different loss functions. We attempted to solve this problem by using the Bregman divergence to provide a unified interpretation of the softmax cross-entropy and negative sampling loss functions. Under this interpretation, we can derive theoretical findings for fair comparison. Experimental results on the FB15k-237 and WN18RR datasets show that the theoretical findings are valid in practical settings.