# Enhancing Short-Text Topic Modeling with LLM-Driven Context Expansion and Prefix-Tuned VAEs

arXiv: https://arxiv.org/abs/2410.03071 · HTML: https://mlsanity.com/paper/2410.03071/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Contrastive Learning for Neural Topic Model](https://mlsanity.com/paper/2110.12764/) (2021)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [Topic Memory Networks for Short Text Classification](https://mlsanity.com/paper/1809.03664/) (2018)
- [vONTSS: vMF based semi-supervised neural topic modeling with optimal transport](https://mlsanity.com/paper/2307.01226/) (2023)
- [DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM](https://mlsanity.com/paper/2310.15296/) (2023)
- [Context Reinforced Neural Topic Modeling over Short Texts](https://mlsanity.com/paper/2008.04545/) (2020)

## Built on by (most cited first, 5 of 5)

- [MALTopic: Multi-Agent LLM Topic Modeling Framework](https://mlsanity.com/paper/2601.15299/) (2026)
- [Understanding Cross-Domain Adaptation in Low-Resource Topic Modeling](https://mlsanity.com/paper/2506.07453/) (2025)
- [Improving Topic Modeling by Distilling Soft Labels from Language Models](https://mlsanity.com/paper/2602.17907/) (2026)
- [Advanced Topic Modeling Techniques for Categorizing Software Vulnerabilities](https://mlsanity.com/paper/2607.03887/) (2026)
- [Domain-Agnostic Neural Topic Modeling with Contextual Token-Level Semantic Graph Representation](https://mlsanity.com/paper/2608.16269/) (2026)

## Abstract (arXiv)

Topic modeling is a powerful technique for uncovering hidden themes within a collection of documents. However, the effectiveness of traditional topic models often relies on sufficient word co-occurrence, which is lacking in short texts. Therefore, existing approaches, whether probabilistic or neural, frequently struggle to extract meaningful patterns from such data, resulting in incoherent topics. To address this challenge, we propose a novel approach that leverages large language models (LLMs) to extend short texts into more detailed sequences before applying topic modeling. To further improve the efficiency and solve the problem of semantic inconsistency from LLM-generated texts, we propose to use prefix tuning to train a smaller language model coupled with a variational autoencoder for short-text topic modeling. Our method significantly improves short-text topic modeling performance, as demonstrated by extensive experiments on real-world datasets with extreme data sparsity, outperforming current state-of-the-art topic models.