# Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

arXiv: https://arxiv.org/abs/2211.12878 · HTML: https://mlsanity.com/paper/2211.12878/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 60 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations](https://mlsanity.com/paper/1805.06201/) (2018)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Analyzing and Improving Representations with the Soft Nearest Neighbor Loss](https://mlsanity.com/paper/1902.01889/) (2019)
- [Contrastive Learning for Label-Efficient Semantic Segmentation](https://mlsanity.com/paper/2012.06985/) (2020)
- [Supporting Clustering with Contrastive Learning](https://mlsanity.com/paper/2103.12953/) (2021)
- [Discovering Discrete Latent Topics with Neural Variational Inference](https://mlsanity.com/paper/1706.00359/) (2017)
- [Probabilistic Latent Semantic Analysis](https://mlsanity.com/paper/1301.6705/) (2013)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Contrastive Learning for Neural Topic Model](https://mlsanity.com/paper/2110.12764/) (2021)
- [Neural Models for Documents with Metadata](https://mlsanity.com/paper/1705.09296/) (2017)
- [Short Text Topic Modeling Techniques, Applications, and Performance: A Survey](https://mlsanity.com/paper/1904.07695/) (2019)

## Built on by (most cited first, 30 of 38)

- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge](https://mlsanity.com/paper/2412.13670/) (2024)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [AKEW: Assessing Knowledge Editing in the Wild](https://mlsanity.com/paper/2402.18909/) (2024)
- [A Survey of Recent Backdoor Attacks and Defenses in Large Language Models](https://mlsanity.com/paper/2406.06852/) (2024)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation](https://mlsanity.com/paper/2410.14425/) (2024)
- [InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling](https://mlsanity.com/paper/2304.03544/) (2023)
- [Towards the TopMost: A Topic Modeling System Toolkit](https://mlsanity.com/paper/2309.06908/) (2023)
- [On the Affinity, Rationality, and Diversity of Hierarchical Topic Modeling](https://mlsanity.com/paper/2401.14113/) (2024)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM](https://mlsanity.com/paper/2310.15296/) (2023)
- [Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion](https://mlsanity.com/paper/2405.17957/) (2024)
- [KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning](https://mlsanity.com/paper/2403.17486/) (2024)
- [Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representation](https://mlsanity.com/paper/2312.02227/) (2023)
- [Topic Modeling as Multi-Objective Contrastive Optimization](https://mlsanity.com/paper/2402.07577/) (2024)
- [Are LLMs Good Zero-Shot Fallacy Classifiers?](https://mlsanity.com/paper/2410.15050/) (2024)
- [A Multi-Modal Contrastive Diffusion Model for Therapeutic Peptide Generation](https://mlsanity.com/paper/2312.15665/) (2023)
- [MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning](https://mlsanity.com/paper/2407.03788/) (2024)
- [Gradient-Boosted Decision Tree for Listwise Context Model in Multimodal Review Helpfulness Prediction](https://mlsanity.com/paper/2305.12678/) (2023)
- [Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detection](https://mlsanity.com/paper/2506.08477/) (2025)
- [Enhancing Short-Text Topic Modeling with LLM-Driven Context Expansion and Prefix-Tuned VAEs](https://mlsanity.com/paper/2410.03071/) (2024)
- [GloCOM: A Short Text Neural Topic Model via Global Clustering Context](https://mlsanity.com/paper/2412.00525/) (2024)
- [Comprehensive Evaluation of Large Language Models for Topic Modeling](https://mlsanity.com/paper/2406.00697/) (2024)
- [Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation](https://mlsanity.com/paper/2501.14166/) (2025)
- [Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation](https://mlsanity.com/paper/2504.13054/) (2025)
- [Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets](https://mlsanity.com/paper/2510.18908/) (2025)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [Graph Contrastive Topic Model](https://mlsanity.com/paper/2307.02078/) (2023)
- [Enhancing Topic Interpretability for Neural Topic Modeling through Topic-wise Contrastive Learning](https://mlsanity.com/paper/2412.17338/) (2024)

## Abstract (arXiv)

To overcome the data sparsity issue in short text topic modeling, existing methods commonly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information. However, most of them do not make full use of the augmented data or the data characteristic: they insufficiently learn the relations among samples in data, leading to dissimilar topic distributions of semantically similar text pairs. To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM). To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient positive and negative sampling strategies based on topic semantics. This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue. Extensive experimental results show that our TSCTM outperforms state-of-the-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions.