# Learning distributed representations with efficient SoftMax normalization

arXiv: https://arxiv.org/abs/2303.17475 · HTML: https://mlsanity.com/paper/2303.17475/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [graph2vec: Learning Distributed Representations of Graphs](https://mlsanity.com/paper/1707.05005/) (2017)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Community detection in networks: A user guide](https://mlsanity.com/paper/1608.00163/) (2016)
- [dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning](https://mlsanity.com/paper/1809.02657/) (2018)
- [GEMSEC: Graph Embedding with Self Clustering](https://mlsanity.com/paper/1802.03997/) (2018)
- [Adaptive Sampled Softmax with Kernel Based Sampling](https://mlsanity.com/paper/1712.00527/) (2017)
- [Sampled Softmax with Random Fourier Features](https://mlsanity.com/paper/1907.10747/) (2019)
- [dna2vec: Consistent vector representations of variable-length k-mers](https://mlsanity.com/paper/1701.06279/) (2017)
- [Simplex2Vec embeddings for community detection in simplicial complexes](https://mlsanity.com/paper/1906.09068/) (2019)
- [Extreme Classification via Adversarial Softmax Approximation](https://mlsanity.com/paper/2002.06298/) (2020)
- [Tweet2Vec: Character-Based Distributed Representations for Social Media](https://mlsanity.com/paper/1605.03481/) (2016)
- [DyLink2Vec: Effective Feature Representation for Link Prediction in Dynamic Networks](https://mlsanity.com/paper/1804.05755/) (2018)
- [Fast Amortized Inference and Learning in Log-linear Models with Randomly Perturbed Nearest Neighbor Search](https://mlsanity.com/paper/1707.03372/) (2017)
- [A spectral method for community detection in moderately-sparse degree-corrected stochastic block models](https://mlsanity.com/paper/1506.08621/) (2015)
- [A unified framework for spectral clustering in sparse graphs](https://mlsanity.com/paper/2003.09198/) (2020)
- [Flexible Multi-layer Sparse Approximations of Matrices and Applications](https://mlsanity.com/paper/1506.07300/) (2015)
- [Revisiting Skip-Gram Negative Sampling Model with Rectification](https://mlsanity.com/paper/1804.00306/) (2018)
- [Improving Negative Sampling for Word Representation using Self-embedded Features](https://mlsanity.com/paper/1710.09805/) (2017)
- [weg2vec: Event embedding for temporal networks](https://mlsanity.com/paper/1911.02425/) (2019)
- [word2vec Skip-Gram with Negative Sampling is a Weighted Logistic PCA](https://mlsanity.com/paper/1705.09755/) (2017)

## Built on by (most cited first, 2 of 2)

- [An embedding-based distance for temporal graphs](https://mlsanity.com/paper/2401.12843/) (2024)
- [A temporal graph dissimilarity measure capturing epidemic spreading behavior](https://mlsanity.com/paper/2610.16456/) (2026)

## Abstract (arXiv)

Learning distributed representations, or embeddings, that encode the relational similarity patterns among objects is a relevant task in machine learning. A popular method to learn the embedding matrices X, Y is optimizing a loss function of the term rm SoftMax(XYT). The complexity required to calculate this term, however, runs quadratically with the problem size, making it a computationally heavy solution. In this article, we propose a linear-time heuristic approximation to compute the normalization constants of rm SoftMax(XYT) for embedding vectors with bounded norms. We show on some pre-trained embedding datasets that the proposed estimation method achieves higher or comparable accuracy with competing methods. From this result, we design an efficient and task-agnostic algorithm that learns the embeddings by optimizing the cross entropy between the softmax and a set of probability distributions given as inputs. The proposed algorithm is interpretable and easily adapted to arbitrary embedding problems. We consider a few use cases and observe similar or higher performances and a lower computational time than similar ``2Vec'' algorithms.