# Distilled Wasserstein Learning for Word Embedding and Topic Modeling

arXiv: https://arxiv.org/abs/1809.04705 · HTML: https://mlsanity.com/paper/1809.04705/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 18 of 18)

- [Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs](https://mlsanity.com/paper/2402.12030/) (2024)
- [Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings](https://mlsanity.com/paper/2203.01570/) (2022)
- [Wasserstein barycenters are NP-hard to compute](https://mlsanity.com/paper/2101.01100/) (2021)
- [vONTSS: vMF based semi-supervised neural topic modeling with optimal transport](https://mlsanity.com/paper/2307.01226/) (2023)
- [Re-evaluating Word Mover's Distance](https://mlsanity.com/paper/2105.14403/) (2021)
- [Fixed Support Tree-Sliced Wasserstein Barycenter](https://mlsanity.com/paper/2109.03431/) (2021)
- [Measure Estimation in the Barycentric Coding Model](https://mlsanity.com/paper/2201.12195/) (2022)
- [Universal Neural Optimal Transport](https://mlsanity.com/paper/2212.00133/) (2022)
- [Knowledge-Aware Bayesian Deep Topic Model](https://mlsanity.com/paper/2209.14228/) (2022)
- [Gromov-Wasserstein-like Distances in the Gaussian Mixture Models Space](https://mlsanity.com/paper/2310.11256/) (2023)
- [KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation](https://mlsanity.com/paper/2307.01878/) (2023)
- [Interpretable contrastive word mover's embedding](https://mlsanity.com/paper/2111.01023/) (2021)
- [TopNet: Learning from Neural Topic Model to Generate Long Stories](https://mlsanity.com/paper/2112.07259/) (2021)
- [Unsupervised Opinion Summarisation in the Wasserstein Space](https://mlsanity.com/paper/2211.14923/) (2022)
- [Hybrid Gromov-Wasserstein Embedding for Capsule Learning](https://mlsanity.com/paper/2209.00232/) (2022)
- [Linearized Wasserstein Barycenters: Synthesis, Analysis, Representational Capacity, and Applications](https://mlsanity.com/paper/2410.23602/) (2024)
- [Exclusive Topic Modeling](https://mlsanity.com/paper/2102.03525/) (2021)
- [Concentrated Document Topic Model](https://mlsanity.com/paper/2102.04449/) (2021)

## Abstract (arXiv)

We propose a novel Wasserstein method with a distillation mechanism, yielding joint learning of word embeddings and topics. The proposed method is based on the fact that the Euclidean distance between word embeddings may be employed as the underlying distance in the Wasserstein topic model. The word distributions of topics, their optimal transports to the word distributions of documents, and the embeddings of words are learned in a unified framework. When learning the topic model, we leverage a distilled underlying distance matrix to update the topic distributions and smoothly calculate the corresponding optimal transports. Such a strategy provides the updating of word embeddings with robust guidance, improving the algorithmic convergence. As an application, we focus on patient admission records, in which the proposed method embeds the codes of diseases and procedures and learns the topics of admissions, obtaining superior performance on clinically-meaningful disease network construction, mortality prediction as a function of admission codes, and procedure recommendation.