mlsanityData to 14 Oct 2026

MMTM: Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion

arXiv 2605.29765 ↗First posted 28 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce MMTM, a modular pipeline for topic discovery in long-form video that integrates speech recognition, audio and visual embeddings, and BERTopic clustering through a deterministic similarity-gated fusion. Evaluated cross-lingually on German (Tagesschau) and English (NBC) broadcast news, joint tri-modal modeling substantially improves topic quality: noise drops from 0.27 to 0.06, transition rate from 0.70 to 0.21, and normalized entropy rises from 0.84 to 0.92, indicating more coherent and temporally stable topics. Cluster validity (Calinski-Harabasz) improves by 5-12X across embedding spaces. Lexical coherence (NPMI) rises from 0.77 to 0.86 on German but is corpus-dependent and does not transfer to the shorter NBC broadcasts. We release the pipeline code and a human-validated 54-hour multimodal video topic corpus with dual-annotator visual evaluation and LLM-assisted labeling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 34 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

15 of 34 references.

References · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  8. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  9. 2017Autoencoding Variational Inference For Topic Modelscited by 169
  10. 2015Neural Variational Inference for Text Processingcited by 111
  11. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
  12. 2024Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approachcited by 12
3 more
  1. 2016Topic Modeling over Short Texts by Incorporating Word Embeddingscited by 9
  2. 2024Neural Multimodal Topic Modeling: A Comprehensive Evaluationcited by 5
  3. 2025VisTopics: A Visual Semantic Unsupervised Approach to Topic Modeling of Video and Image Datacited by 1

Built on by

Not cited yet.