# CEMTM: Contextual Embedding-based Multimodal Topic Modeling

arXiv: https://arxiv.org/abs/2509.11465 · HTML: https://mlsanity.com/paper/2509.11465/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 14 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [Cross-lingual Contextualized Topic Models with Zero-shot Learning](https://mlsanity.com/paper/2004.07737/) (2020)
- [Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling](https://mlsanity.com/paper/2403.16248/) (2024)
- [Specialising Word Vectors for Lexical Entailment](https://mlsanity.com/paper/1710.06371/) (2017)
- [A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding](https://mlsanity.com/paper/2305.03668/) (2023)
- [CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling](https://mlsanity.com/paper/2305.09329/) (2023)
- [Neural Attention-Aware Hierarchical Topic Model](https://mlsanity.com/paper/2110.07161/) (2021)
- [Multilingual and Multimodal Topic Modelling with Pretrained Embeddings](https://mlsanity.com/paper/2211.08057/) (2022)
- [PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models](https://mlsanity.com/paper/2312.06093/) (2023)
- [Neural Multimodal Topic Modeling: A Comprehensive Evaluation](https://mlsanity.com/paper/2403.17308/) (2024)

## Built on by (most cited first, 4 of 4)

- [Sparse Autoencoders are Topic Models](https://mlsanity.com/paper/2511.16309/) (2025)
- [Improving Topic Modeling by Distilling Soft Labels from Language Models](https://mlsanity.com/paper/2602.17907/) (2026)
- [Beyond Semantics: Modeling Factual and Affective Perceptual Experiences from Vision-Language Data](https://mlsanity.com/paper/2606.03345/) (2026)
- [MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval](https://mlsanity.com/paper/2606.18508/) (2026)

## Abstract (arXiv)

We introduce CEMTM, a context-enhanced multimodal topic model designed to infer coherent and interpretable topic structures from both short and long documents containing text and images. CEMTM builds on fine-tuned large vision language models (LVLMs) to obtain contextualized embeddings, and employs a distributional attention mechanism to weight token-level contributions to topic inference. A reconstruction objective aligns topic-based representations with the document embedding, encouraging semantic consistency across modalities. Unlike existing approaches, CEMTM can process multiple images per document without repeated encoding and maintains interpretability through explicit word-topic and document-topic distributions. Extensive experiments on six multimodal benchmarks show that CEMTM consistently outperforms unimodal and multimodal baselines, achieving a remarkable average LLM score of 2.61. Further analysis shows its effectiveness in downstream few-shot retrieval and its ability to capture visually grounded semantics in complex domains such as scientific articles.