# FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model

arXiv: https://arxiv.org/abs/2405.17978 · HTML: https://mlsanity.com/paper/2405.17978/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 91 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Top2Vec: Distributed Representations of Topics](https://mlsanity.com/paper/2008.09470/) (2020)
- [Improving GANs Using Optimal Transport](https://mlsanity.com/paper/1803.05573/) (2018)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Discovering Discrete Latent Topics with Neural Variational Inference](https://mlsanity.com/paper/1706.00359/) (2017)
- [Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence](https://mlsanity.com/paper/2107.02173/) (2021)
- [Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too!](https://mlsanity.com/paper/2004.14914/) (2020)
- [Probabilistic Latent Semantic Analysis](https://mlsanity.com/paper/1301.6705/) (2013)
- [Topic Modelling Meets Deep Neural Networks: A Survey](https://mlsanity.com/paper/2103.00498/) (2021)
- [Deep k-Means: Jointly clustering with k-Means and learning representations](https://mlsanity.com/paper/1806.10069/) (2018)
- [Neural Topic Model via Optimal Transport](https://mlsanity.com/paper/2008.13537/) (2020)
- [Effective Neural Topic Modeling with Embedding Clustering Regularization](https://mlsanity.com/paper/2306.04217/) (2023)
- [The Dynamic Embedded Topic Model](https://mlsanity.com/paper/1907.05545/) (2019)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings](https://mlsanity.com/paper/2203.01570/) (2022)
- [Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics](https://mlsanity.com/paper/2204.09874/) (2022)

## Built on by (most cited first, 30 of 35)

- [AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge](https://mlsanity.com/paper/2412.13670/) (2024)
- [Are LLMs Good Zero-Shot Fallacy Classifiers?](https://mlsanity.com/paper/2410.15050/) (2024)
- [Encoding and Controlling Global Semantics for Long-form Video Question Answering](https://mlsanity.com/paper/2405.19723/) (2024)
- [Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders](https://mlsanity.com/paper/2507.23220/) (2025)
- [UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning](https://mlsanity.com/paper/2502.15082/) (2025)
- [CutPaste&Find: Efficient Multimodal Hallucination Detector with Visual-aid Knowledge Base](https://mlsanity.com/paper/2502.12591/) (2025)
- [Probing Association Biases in LLM Moderation Over-Sensitivity](https://mlsanity.com/paper/2505.23914/) (2025)
- [ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control](https://mlsanity.com/paper/2510.18232/) (2025)
- [Improving Topic Modeling of Social Media Short Texts with Rephrasing: A Case Study of COVID-19 Related Tweets](https://mlsanity.com/paper/2510.18908/) (2025)
- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [Personalized Benchmarking: Evaluating LLMs by Individual Preferences](https://mlsanity.com/paper/2604.18943/) (2026)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [Poisson-Process Topic Model for Integrating Knowledge from Pre-trained Language Models](https://mlsanity.com/paper/2503.17809/) (2025)
- [Merging Embedded Topics with Optimal Transport for Online Topic Modeling on Data Streams](https://mlsanity.com/paper/2504.07711/) (2025)
- [E-LDA: Toward Interpretable LDA Topic Models with Strong Guarantees in Logarithmic Parallel Time](https://mlsanity.com/paper/2506.07747/) (2025)
- [More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answering](https://mlsanity.com/paper/2511.20086/) (2025)
- [Hierarchical Graph Topic Modeling with Topic Tree-based Transformer](https://mlsanity.com/paper/2502.11345/) (2025)
- [topicwizard -- a Modern, Model-agnostic Framework for Topic Model Visualization and Interpretation](https://mlsanity.com/paper/2505.13034/) (2025)
- [Constrained Non-negative Matrix Factorization for Guided Topic Modeling of Minority Topics](https://mlsanity.com/paper/2505.16493/) (2025)
- [Understanding Cross-Domain Adaptation in Low-Resource Topic Modeling](https://mlsanity.com/paper/2506.07453/) (2025)
- [LLM as Attention-Informed NTM and Topic Modeling as long-input Generation: Interpretability and long-Context Capability](https://mlsanity.com/paper/2510.03174/) (2025)
- [Meta-probabilistic Modeling](https://mlsanity.com/paper/2601.04462/) (2026)
- [A Neural Topic Method Using a Large-Language-Model-in-the-Loop for Business Research](https://mlsanity.com/paper/2603.03623/) (2026)
- [CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents](https://mlsanity.com/paper/2603.03884/) (2026)
- [Disentangling Similarity and Relatedness in Topic Models](https://mlsanity.com/paper/2603.10619/) (2026)
- [PRISM: PRIor from corpus Statistics for topic Modeling](https://mlsanity.com/paper/2603.29406/) (2026)
- [CobwebTM: Probabilistic Concept Formation for Lifelong and Hierarchical Topic Modeling](https://mlsanity.com/paper/2604.14489/) (2026)
- [A comparative study of transformer-based embeddings for topic coherence](https://mlsanity.com/paper/2605.28832/) (2026)
- [TGCM: Topic-Guided Consistency Modeling for One-Step Disentanglement of Interleaved APT Technique Sequences](https://mlsanity.com/paper/2606.18651/) (2026)
- [Label Semantic Expansion via Label Guided Neural Topic Modeling](https://mlsanity.com/paper/2608.30216/) (2026)

## Abstract (arXiv)

Topic models have been evolving rapidly over the years, from conventional to recent neural models. However, existing topic models generally struggle with either effectiveness, efficiency, or stability, highly impeding their practical applications. In this paper, we propose FASTopic, a fast, adaptive, stable, and transferable topic model. FASTopic follows a new paradigm: Dual Semantic-relation Reconstruction (DSR). Instead of previous conventional, VAE-based, or clustering-based methods, DSR directly models the semantic relations among document embeddings from a pretrained Transformer and learnable topic and word embeddings. By reconstructing through these semantic relations, DSR discovers latent topics. This brings about a neat and efficient topic modeling framework. We further propose a novel Embedding Transport Plan (ETP) method. Rather than early straightforward approaches, ETP explicitly regularizes the semantic relations as optimal transport plans. This addresses the relation bias issue and thus leads to effective topic modeling. Extensive experiments on benchmark datasets demonstrate that our FASTopic shows superior effectiveness, efficiency, adaptivity, stability, and transferability, compared to state-of-the-art baselines across various scenarios.