# Enhancing BERTopic with Intermediate Layer Representations

arXiv: https://arxiv.org/abs/2505.06696 · HTML: https://mlsanity.com/paper/2505.06696/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 10 May 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Topic Modeling Based Multi-modal Depression Detection](https://mlsanity.com/paper/1803.10384/) (2018)
- [Universal Text Representation from BERT: An Empirical Study](https://mlsanity.com/paper/1910.07973/) (2019)
- [ChatGPT in education: A discourse analysis of worries and concerns on social media](https://mlsanity.com/paper/2305.02201/) (2023)
- [Comparison of Topic Modelling Approaches in the Banking Context](https://mlsanity.com/paper/2402.03176/) (2024)
- [Topic Modelling on Consumer Financial Protection Bureau Data: An Approach Using BERT Based Embeddings](https://mlsanity.com/paper/2205.07259/) (2022)

## Abstract (arXiv)

BERTopic is a topic modeling algorithm that leverages transformer-based embeddings to create dense clusters, enabling the estimation of topic structures and the extraction of valuable insights from a corpus of documents. This approach allows users to efficiently process large-scale text data and gain meaningful insights into its structure. While BERTopic is a powerful tool, embedding preparation can vary, including extracting representations from intermediate model layers and applying transformations to these embeddings. In this study, we evaluate 18 different embedding representations and present findings based on experiments conducted on three diverse datasets. To assess the algorithm's performance, we report topic coherence and topic diversity metrics across all experiments. Our results demonstrate that, for each dataset, it is possible to find an embedding configuration that performs better than the default setting of BERTopic. Additionally, we investigate the influence of stop words on different embedding configurations.