mlsanityData to 13 Oct 2026

Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence

arXiv 2004.03974 ↗First posted 8 Apr 2020cs.CLcited by 121

Not on the map. Why?

Abstract (arXiv)

Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.

Key facts. Cited by 121 papers in the map (data through 13 Oct 2026). First posted on arXiv 8 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 13; 2022: 21; 2023: 24; 2024: 18; 2025: 34; 2026: 11.

121 papers.

Sort:

By area:

  1. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 792
  2. 2021Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherencecited by 74
  3. 2021Topic Modelling Meets Deep Neural Networks: A Surveycited by 51
  4. 2022Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topicscited by 35
  5. 2022Topic Discovery via Latent Space Clustering of Pretrained Language Model Representationscited by 34
  6. 2024FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Modelcited by 34
  7. 2021Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapterscited by 23
  8. 2023Towards the TopMost: A Topic Modeling System Toolkitcited by 21
  9. 2023Contextualized Topic Coherence Metricscited by 19
  10. 2024A Survey on Neural Topic Models: Methods, Applications, and Challengescited by 18
  11. 2022Sentence-level Privacy for Document Embeddingscited by 16
  12. 2023vONTSS: vMF based semi-supervised neural topic modeling with optimal transportcited by 16
38 more
  1. 2023DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLMcited by 16
  2. 2022HyperMiner: Topic Taxonomy Mining with Hyperbolic Embeddingcited by 15
  3. 2022Effective Seed-Guided Topic Discovery by Integrating Multiple Types of Contextscited by 12
  4. 2024Topic Modeling as Multi-Objective Contrastive Optimizationcited by 12
  5. 2025HICode: Hierarchical Inductive Coding with LLMscited by 12
  6. 2022Are Neural Topic Models Broken?cited by 11
  7. 2022Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian Mediacited by 10
  8. 2024Neural Topic Modeling with Large Language Models in the Loopcited by 10
  9. 2024NeuroMax: Enhancing Neural Topic Modeling via Maximizing Mutual Information and Group Topic Regularizationcited by 9
  10. 2021Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modelingcited by 8
  11. 2022Conversational Speech Recognition By Learning Conversation-level Characteristicscited by 8
  12. 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
  13. 2022Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASRcited by 7
  14. 2023CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modelingcited by 7
  15. 2025Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoderscited by 7
  16. 2021Neural Attention-Aware Hierarchical Topic Modelcited by 6
  17. 2022Multilingual and Multimodal Topic Modelling with Pretrained Embeddingscited by 6
  18. 2022Knowledge-Aware Bayesian Deep Topic Modelcited by 5
  19. 2023Diversity-Aware Coherence Loss for Improving Neural Topic Modelscited by 5
  20. 2024Neural Multimodal Topic Modeling: A Comprehensive Evaluationcited by 5
  21. 2024Enhancing Short-Text Topic Modeling with LLM-Driven Context Expansion and Prefix-Tuned VAEscited by 5
  22. 2023Improving Contextualized Topic Models with Negative Samplingcited by 4
  23. 2024GINopic: Topic Modeling with Graph Isomorphism Networkcited by 4
  24. 2024Qualitative Insights Tool (QualIT): LLM Enhanced Topic Modelingcited by 4
  25. 2024GloCOM: A Short Text Neural Topic Model via Global Clustering Contextcited by 4
  26. 2025Bridging the Evaluation Gap: Leveraging Large Language Models for Topic Model Evaluationcited by 4
  27. 2025CEMTM: Contextual Embedding-based Multimodal Topic Modelingcited by 4
  28. 2021Evaluation of Thematic Coherence in Microblogscited by 3
  29. 2021Artificial intelligence for Sustainable Energy: A Contextual Topic Modeling and Content Analysiscited by 3
  30. 2022Topic Modeling on Podcast Short-Text Metadatacited by 3
  31. 2022Unsupervised Audio-Visual Lecture Segmentationcited by 3
  32. 2022Human in the loop: How to effectively create coherent topics by manually labeling only a few documents per classcited by 3
  33. 2023Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networkscited by 3
  34. 2023Do Neural Topic Models Really Need Dropout? Analysis of the Effect of Dropout in Topic Modelingcited by 3
  35. 2025ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clusteringcited by 3
  36. 2025XTRA: Cross-Lingual Topic Modeling with Topic and Representation Alignmentscited by 3
  37. 2025Sparse Autoencoders are Topic Modelscited by 3
  38. 2021Detecting Polarized Topics Using Partisanship-aware Contextualized Topic Embeddingscited by 2

50 most cited of 121 All 121 (JSON)