# Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

arXiv: https://arxiv.org/abs/2203.01570 · HTML: https://mlsanity.com/paper/2203.01570/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Learning with a Wasserstein Loss](https://mlsanity.com/paper/1506.05439/) (2015)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [A Prototype-Oriented Framework for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2110.12024/) (2021)
- [Hierarchical Optimal Transport for Document Representation](https://mlsanity.com/paper/1906.10827/) (2019)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Neural Topic Model via Optimal Transport](https://mlsanity.com/paper/2008.13537/) (2020)
- [Neural Models for Documents with Metadata](https://mlsanity.com/paper/1705.09296/) (2017)
- [WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling](https://mlsanity.com/paper/1803.01328/) (2018)
- [Improving Topic Models with Latent Feature Word Representations](https://mlsanity.com/paper/1810.06306/) (2018)
- [Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network](https://mlsanity.com/paper/2107.02757/) (2021)
- [Gamma Belief Networks](https://mlsanity.com/paper/1512.03081/) (2015)
- [Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec](https://mlsanity.com/paper/1605.02019/) (2016)
- [Distilled Wasserstein Learning for Word Embedding and Topic Modeling](https://mlsanity.com/paper/1809.04705/) (2018)
- [TopicNet: Semantic Graph-Guided Topic Discovery](https://mlsanity.com/paper/2110.14286/) (2021)
- [Jointly Learning Word Embeddings and Latent Topics](https://mlsanity.com/paper/1706.07276/) (2017)
- [MetaLDA: a Topic Model that Efficiently Incorporates Meta information](https://mlsanity.com/paper/1709.06365/) (2017)
- [Neural Embedding Allocation: Distributed Representations of Topic Models](https://mlsanity.com/paper/1909.04702/) (2019)

## Built on by (most cited first, 30 of 36)

- [Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2310.14228/) (2023)
- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Transformed Distribution Matching for Missing Value Imputation](https://mlsanity.com/paper/2302.10363/) (2023)
- [POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models](https://mlsanity.com/paper/2305.00350/) (2023)
- [Tuning Multi-mode Token-level Prompt Alignment across Modalities](https://mlsanity.com/paper/2309.13847/) (2023)
- [Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification](https://mlsanity.com/paper/2208.02951/) (2022)
- [PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification](https://mlsanity.com/paper/2307.09066/) (2023)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [vONTSS: vMF based semi-supervised neural topic modeling with optimal transport](https://mlsanity.com/paper/2307.01226/) (2023)
- [HyperMiner: Topic Taxonomy Mining with Hyperbolic Embedding](https://mlsanity.com/paper/2210.10625/) (2022)
- [Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport](https://mlsanity.com/paper/2210.04144/) (2022)
- [Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2303.09100/) (2023)
- [Topic Modeling as Multi-Objective Contrastive Optimization](https://mlsanity.com/paper/2402.07577/) (2024)
- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [Neural Topic Modeling with Large Language Models in the Loop](https://mlsanity.com/paper/2411.08534/) (2024)
- [NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization](https://mlsanity.com/paper/2409.19749/) (2024)
- [Optimal Transport for Structure Learning Under Missing Data](https://mlsanity.com/paper/2402.15255/) (2024)
- [TsCA: On the Semantic Consistency Alignment via Conditional Transport for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2408.08703/) (2024)
- [Knowledge-Aware Bayesian Deep Topic Model](https://mlsanity.com/paper/2209.14228/) (2022)
- [Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation](https://mlsanity.com/paper/2312.11532/) (2023)
- [GloCOM: A Short Text Neural Topic Model via Global Clustering Context](https://mlsanity.com/paper/2412.00525/) (2024)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [A Prototype-Oriented Clustering for Domain Shift with Source Privacy](https://mlsanity.com/paper/2302.03807/) (2023)
- [Topic Modeling with Fine-tuning LLMs and Bag of Sentences](https://mlsanity.com/paper/2408.03099/) (2024)
- [Improving Sparse Autoencoder with Dynamic Attention](https://mlsanity.com/paper/2604.14925/) (2026)
- [Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences](https://mlsanity.com/paper/2302.03106/) (2023)
- [Parameter Estimation in DAGs from Incomplete Data via Optimal Transport](https://mlsanity.com/paper/2305.15927/) (2023)
- [Variational Language Concepts for Interpreting Foundation Language Models](https://mlsanity.com/paper/2410.03964/) (2024)

## Abstract (arXiv)

A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. It is focused on capturing the word co-occurrences in a document and hence often suffers from poor performance in analyzing short documents. In addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffers from large approximation error. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations.