mlsanityData to 7 Oct 2026

Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding

arXiv 2106.07250 ↗First posted 14 Jun 2021cs.LG cs.CL stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

In knowledge graph embedding, the theoretical relationship between the softmax cross-entropy and negative sampling loss functions has not been investigated. This makes it difficult to fairly compare the results of the two different loss functions. We attempted to solve this problem by using the Bregman divergence to provide a unified interpretation of the softmax cross-entropy and negative sampling loss functions. Under this interpretation, we can derive theoretical findings for fair comparison. Experimental results on the FB15k-237 and WN18RR datasets show that the theoretical findings are valid in practical settings.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

21 of 37 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 929
  5. 2016Complex Embeddings for Simple Link Predictioncited by 897
  6. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  7. 2017Convolutional 2D Knowledge Graph Embeddingscited by 729
  8. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  9. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  10. 2019TuckER: Tensor Factorization for Knowledge Graph Completioncited by 290
  11. 2020Does label smoothing mitigate label noise?cited by 204
  12. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 109
9 more
  1. 2020PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddingscited by 77
  2. 2019Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modelingcited by 71
  3. 2020Understanding Negative Sampling in Graph Representation Learningcited by 66
  4. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  5. 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 40
  6. 2017Knowledge Base Completion: Baselines Strike Backcited by 39
  7. 2014Notes on Noise Contrastive Estimation and Negative Samplingcited by 26
  8. 2018Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processingcited by 15
  9. 2017A Simple Language Model based on PMI Matrix Approximationscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 3.

7 papers.

  1. 2022Comprehensive Analysis of Negative Sampling in Knowledge Graph Representation Learningcited by 12
  2. 2024Towards Artwork Explanation in Large-scale Vision Language Modelscited by 8
  3. 2024Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Modelscited by 7
  4. 2022Subsampling for Knowledge Graph Embedding Explainedcited by 2
  5. 2023Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?cited by 1
  6. 2023Model-based Subsampling for Knowledge Graph Completioncited by 0
  7. 2024Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embeddingcited by 0