mlsanityData to 7 Oct 2026

Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances

arXiv 2405.12775 ↗First posted 21 May 2024cs.MM cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions. Existing methods manifest limitations in leveraging nonverbal information for discerning complex semantics in unsupervised scenarios. This paper introduces a novel unsupervised multimodal clustering method (UMC), making a pioneering contribution to this field. UMC introduces a unique approach to constructing augmentation views for multimodal data, which are then used to perform pre-training to establish well-initialized representations for subsequent clustering. An innovative strategy is proposed to dynamically select high-quality samples as guidance for representation learning, gauged by the density of each sample's nearest neighbors. Besides, it is equipped to automatically determine the optimal value for the top-K parameter in each cluster to refine sample selection. Finally, both high- and low-quality samples are used to learn representations conducive to effective clustering. We build baselines on benchmark multimodal intent and dialogue act datasets. UMC shows remarkable improvements of 2-7% scores in clustering metrics over state-of-the-art methods, marking the first successful endeavor in this domain. The complete code and data are available at https://github.com/thuiar/UMC.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 72 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

30 of 72 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  12. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
18 more
  1. 2020Contrastive Clusteringcited by 264
  2. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  3. 2016Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clusteringcited by 237
  4. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  5. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  6. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  7. 2021Supporting Clustering with Contrastive Learningcited by 90
  8. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  9. 2020Discovering New Intents with Deep Aligned Clusteringcited by 68
  10. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  11. 2019Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinementcited by 56
  12. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  13. 2024MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversationscited by 25
  14. 2021TEXTOIR: An Integrated and Visualized Platform for Text Open Intent Recognitioncited by 24
  15. 2023Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognitioncited by 18
  16. 2023A Clustering Framework for Unsupervised and Semi-supervised New Intent Discoverycited by 9
  17. 2023A Diffusion Weighted Graph Framework for New Intent Discoverycited by 7
  18. 2023Decoupling Pseudo Label Disambiguation and Representation Learning for Generalized Intent Discoverycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  2. 2025MultiCheck: Strengthening Web Trust with Unified Multimodal Fact Verificationcited by 1
  3. 2026Dual-Learning based Penalized Multi-Align Clustering for Multi-View Incomplete and Disorderly Datacited by 0