# Poisson-Process Topic Model for Integrating Knowledge from Pre-trained Language Models

arXiv: https://arxiv.org/abs/2503.17809 · HTML: https://mlsanity.com/paper/2503.17809/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 22 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Cross-lingual Contextualized Topic Models with Zero-shot Learning](https://mlsanity.com/paper/2004.07737/) (2020)
- [Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning](https://mlsanity.com/paper/2211.12878/) (2022)
- [FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model](https://mlsanity.com/paper/2405.17978/) (2024)
- [Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations](https://mlsanity.com/paper/2202.04582/) (2022)
- [A Survey on Neural Topic Models: Methods, Applications, and Challenges](https://mlsanity.com/paper/2401.15351/) (2024)
- [Using SVD for Topic Modeling](https://mlsanity.com/paper/1704.07016/) (2017)
- [Co-citation and Co-authorship Networks of Statisticians](https://mlsanity.com/paper/2204.11194/) (2022)
- [NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularization](https://mlsanity.com/paper/2409.19749/) (2024)
- [Pairwise Covariates-adjusted Block Model for Community Detection](https://mlsanity.com/paper/1807.03469/) (2018)
- [A fast algorithm with minimax optimal guarantees for topic models with an unknown number of topics](https://mlsanity.com/paper/1805.06837/) (2018)
- [Recent Advances in Text Analysis](https://mlsanity.com/paper/2401.00775/) (2024)
- [GloCOM: A Short Text Neural Topic Model via Global Clustering Context](https://mlsanity.com/paper/2412.00525/) (2024)

## Built on by (most cited first, 1 of 1)

- [Optimal Demixing of Nonparametric Densities](https://mlsanity.com/paper/2603.27457/) (2026)

## Abstract (arXiv)

Topic modeling is traditionally applied to word counts without accounting for the context in which words appear. Recent advancements in large language models (LLMs) offer contextualized word embeddings, which capture deeper meaning and relationships between words. We aim to leverage such embeddings to improve topic modeling. We use a pre-trained LLM to convert each document into a sequence of word embeddings. This sequence is then modeled as a Poisson point process, with its intensity measure expressed as a convex combination of K base measures, each corresponding to a topic. To estimate these topics, we propose a flexible algorithm that integrates traditional topic modeling methods, enhanced by net-rounding applied before and kernel smoothing applied after. One advantage of this framework is that it treats the LLM as a black box, requiring no fine-tuning of its parameters. Another advantage is its ability to seamlessly integrate any traditional topic modeling approach as a plug-in module, without the need for modifications Assuming each topic is a β-H"older smooth intensity measure on the embedded space, we establish the rate of convergence of our method. We also provide a minimax lower bound and show that the rate of our method matches with the lower bound when β≤ 1. Additionally, we apply our method to several datasets, providing evidence that it offers an advantage over traditional topic modeling approaches.